How a new content offer with strengthened value prop in a dear reader impacted email acquisition
The Gospel Coalition
Experiment Summary
Timeframe: 11/21/2022 - 12/02/2022
When running an in-site content offer, we were seeing decent response rates from web visitors but we were curious about the lifespan of having a single offer available. This free eBook had been available and marketed to the audience for several months, so at some point returning visitors to the site would no longer engage if they had already received the offer. We had a new content offer to launch, but rather than replacing the existing content offer we wanted to understand how the audience would respond comparatively. We split the traffic 50/50 between the two offers to determine how this change would impact both email acquisition and instant donor conversion before rolling out the new offer to 100% of traffic.
Research Question
We believe that highlighting a new content offer for website visitors will achieve an increase in email acquisition.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Secular Creed Dear Reader | 0.06% | $0.00 | ||
| T1: | New City Catechism Dear Reader | 1.0% | 1,698.2% | 100.0% | $0.00 |
This experiment has a required sample size of 448 in order to be valid. Since the experiment had a total sample size of 199,579, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 1,698.2% increase in conversion rate
× 0% increase in average gift
Key Learnings
What we found after running the experiment for one week, was that the treatment—a new devotional eBook—achieved a 1,698% increase in email acquisition rate with 100% level of confidence. We were surprised to see this increase was not limited to returning visitors, but new visitors saw similar lifts as well. Traffic from email and social and both desktop and mobile all saw valid increases in email acquisition. Donations also experienced a 51% increase with 88% level of confidence. We believe this increase is partly due to our hypothesis that a new offer would be more appealing than an offer visitors could have already seen multiple times. But we also believe the new offer was presented with an increased sense of relevancy and urgency that strengthened the value proposition and made it more appealing for visitors to get. The copy encouraged visitors to get their copy now in the last remaining weeks of the year so they could begin the devotional in the new year which was quickly approaching. This specificity on the timeline helps to answer the question in the mind of the visitors – why should I get this offer TODAY, rather than any other day or not at all? This relevance/urgency is an element missing from the control that could explain why we saw such a significant lift in results.