How a revised headline and CTA impacted signups for a weekly newsletter
The Gospel Coalition
Experiment Summary
Timeframe: 05/04/2022 - 06/06/2022
Our third test on this page had a significant variance in the sample size on the control versus the treatment so we wanted to relaunch the test to make sure the results were not muddied. We split traffic between the control page and a treatment page with a headline that had a stronger emphasis on the VALUE provided to the subscriber rather than the newsletter product itself: “Get weekly access to trusted, timely, gospel-centered resources.” We also created a treatment to the button copy to provide additional clarity in the call to action:“Send TGC Weekly to my inbox »”
Research Question
We believe that increasing the value and clarity in an email newsletter signup form for potential newsletter subscribers will achieve an increase in email acquisition.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.00% | ||
| T1: | Headline/Button Copy | 65.9% | 100.0% | 100.0% |
This experiment has a required sample size of 4 in order to be valid. Since the experiment had a total sample size of 45, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 100.0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The treatment ran for one month and we observed a 24.2% decrease in conversion rate with 97.7% level of confidence so the results are valid. This time our sample sizes were accurately proportionate so these results override the increase we observed on the previous test. Our hypothesis is that getting TGC’s “best content” was a more appealing description and “trusted, timely, gospel-centered resources” did not provide enough clarity, appeal, or value to convince users to subscribe to the weekly newsletter.
Observations of the impact on other visitor segments in the experiment:
- Desktop visitors had a 33.2% decrease in email acquisitions with a 95% level of confidence.
- Returning visitors had a 40.2% decrease in email acquisitions with a 98% level of confidence.
- Email visitors had a 28.6% decrease in email acquisitions with a 100% level of confidence.