Helpful language vs admonishment on a devotional landing page Experiment ID: #18752

Care Net

Experiment Summary

Timeframe: 10/16/2019 - 11/6/2019

After our test on the Defending Your Pro-Life Beliefs course page where we found that conversational language directed at helping visitors outperformed more militant “Learn to defend your pro-life beliefs” language, we wanted to try a similar approach to our successful 40-Day devotional Prayers for Life. Our control version focused on the need to be more like Jesus and prayerfully interceding for the unborn at risk of abortion. The control copy focused on how we “can’t afford” to stop praying and used heavier language to make visitors feel obligated to pray, and hopefully, download the devotional.

Our variation used language directed at providing a “daily reminder” to pray to a person’s inbox. The new copy assumed that visitors were already praying for the unborn, rather than assuming that they weren’t.

We tested this new approach on both the landing page and in the ad.

Research Question

Will more conversational and helpful copy outperform  spiritually obligatory language on our devotional landing page?

Design

C: Control
T1: Helpful Language

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Control 41.6%
T1: Helpful Language 64.9% 55.9% 100.0%

This experiment has a required sample size of 35 in order to be valid. Since the experiment had a total sample size of 725, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 55.9% increase in conversion rate
× 0% increase in average gift

Key Learnings

During the entirety of the 3-week experiment, the new language outperformed the old. We decided to use Unbounce’s “smart traffic” feature rather than a straight A/B test for this experiment and the editor showed that this approach delivered an additional 40 signups versus just picking a single “winner” during the 3-week test. While the results had not yet reached 95% certainty, the consistent improvement of the new language coupled with the prior testing of similar language on the Defending your pro-life beliefs course which also performed better, caused us to choose a winner.


Experiment Documented by...

Scoreboard NextAfter

Scoreboard is . If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.