The Gospel Coalition

How a revised headline and CTA impacted signups for a weekly newsletter

Experiment ID: #93535

The Gospel Coalition

Experiment Summary

Timeframe: 05/04/2022 - 06/06/2022

Our third test on this page had a significant variance in the sample size on the control versus the treatment so we wanted to relaunch the test to make sure the results were not muddied. We split traffic between the control page and a treatment page with a headline that had a stronger emphasis on the VALUE provided to the subscriber rather than the newsletter product itself: “Get weekly access to trusted, timely, gospel-centered resources.” We also created a treatment to the button copy to provide additional clarity in the call to action:“Send TGC Weekly to my inbox »”

Research Question

We believe that increasing the value and clarity in an email newsletter signup form for potential newsletter subscribers will achieve an increase in email acquisition.

Design

C: Control
T1: Headline/Button Copy

Results

  Treatment Name Conv. Rate Relative Difference Confidence
C: Control 0.00%
T1: Headline/Button Copy 65.9% 100.0% 100.0%

This experiment has a required sample size of 4 in order to be valid. Since the experiment had a total sample size of 45, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 100.0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The treatment ran for one month and we observed a 24.2% decrease in conversion rate with 97.7% level of confidence so the results are valid. This time our sample sizes were accurately proportionate so these results override the increase we observed on the previous test. Our hypothesis is that getting TGC’s “best content” was a more appealing description and “trusted, timely, gospel-centered resources” did not provide enough clarity, appeal, or value to convince users to subscribe to the weekly newsletter.

Observations of the impact on other visitor segments in the experiment:

  • Desktop visitors had a 33.2% decrease in email acquisitions with a 95% level of confidence.
  • Returning visitors had a 40.2% decrease in email acquisitions with a 98% level of confidence.
  • Email visitors had a 28.6% decrease in email acquisitions with a 100% level of confidence.


Experiment Documented by Rebekah Josefy
Rebekah Josefy is an Optimization Director at NextAfter.

Question about experiment #93535

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.