How a concealing headline affects email acquisition rate Experiment ID: #8140

Hoover Institution

Experiment Summary

Timeframe: 12/8/2017 - 12/18/2017

The Hoover Institution recently launched a new eBook offer titled, ‘Make America Exceptional Again.’ As they began to advertise this offer on Facebook, they wondered: Will a headline that teases the central issue of the eBook increase email acquisition? They set up a treatment page and launched an A/B test to find out.

Research Question

Will a headline that teases the central issue of an offer increase email acquisition rate?


C: Control
T1: Treatment 1


Treatment Name Conv. Rate Relative Difference Confidence
C: Control 41.3%
T1: Treatment 1 54.0% 30.8% 99.1%

This experiment has a required sample size of 118 in order to be valid. Since the experiment had a total sample size of 419, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 30.8% increase in conversion rate
× 0% increase in average gift

Key Learnings

The Hoover Institution found that this treatment version did, in fact, increase email acquisition rate by 30%. It’s clear the second headline increase page visitor motivation. As we look at the difference between these two headlines, we can see that the first headline promises to offer a solution to a very specific problem. The second headline, however, focuses on a broader, more concealed problem–as if the reader was being let in on a secret. It also doesn’t offer an immediate solution.

For these reasons, it’s likely that the visitors who visited the second version of the page felt more motivation to ‘find out’ the answer. Further testing will be conducted on this page, as well as the second ‘Thank you’ page to help improve overall offer performance.

Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.