How the right offer affects emails acquired

Experiment ID: #11717


Experiment Summary

Timeframe: 10/28/2015 - 11/10/2015

When we first launched the new website, we had some internal debate about the best offer to use in the footer of the site. Historically, we had offered a subscription to our weekly newsletter, This vs. That, which allowed subscribers to receive one experiment per week from the Research Library.

Since this was our main website, we hypothesized that maybe the more general offer of “Staying Informed” would work better for the visitors. We decided to set up a test that would allow us to evaluate these two different offers.

Research Question

Which offer in the footer of the NextAfter website will lead to the greatest number of emails acquired?


C: This vs. That
T1: Stay Informed


 Treatment NameConv. RateRelative DifferenceConfidence
C: This vs. That 0.00%
T1: Stay Informed 4.1%100.0% 98.8%

This experiment has a required sample size of 85 in order to be valid. Since the experiment had a total sample size of 321, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 100.0% increase in conversion rate
× 0% increase in average gift

Key Learnings

We discovered that the “Stay Informed” offer led to an infinitely better number of emails captured (none versus some is by definition infinitely better). While the This vs. That newsletter is still one of our more popular offers, a more general contact form was a better fit given the motivation of the people visiting the website.

Experiment Documented by Kevin Peters

Question about experiment #11717

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.