How a concealing headline affects email acquisition rate
Hoover Institution
Experiment Summary
Timeframe: 12/08/2017 - 12/18/2017
The Hoover Institution recently launched a new eBook offer titled, ‘Make America Exceptional Again.’ As they began to advertise this offer on Facebook, they wondered: Will a headline that teases the central issue of the eBook increase email acquisition? They set up a treatment page and launched an A/B test to find out.
Research Question
Will a headline that teases the central issue of an offer increase email acquisition rate?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 41.3% | ||
| T1: | Treatment 1 | 54.0% | 30.8% | 99.1% |
This experiment has a required sample size of 118 in order to be valid. Since the experiment had a total sample size of 419, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 30.8% increase in conversion rate
× 0% increase in average gift
Key Learnings
The Hoover Institution found that this treatment version did, in fact, increase email acquisition rate by 30%. It’s clear the second headline increase page visitor motivation. As we look at the difference between these two headlines, we can see that the first headline promises to offer a solution to a very specific problem. The second headline, however, focuses on a broader, more concealed problem–as if the reader was being let in on a secret. It also doesn’t offer an immediate solution.
For these reasons, it’s likely that the visitors who visited the second version of the page felt more motivation to ‘find out’ the answer. Further testing will be conducted on this page, as well as the second ‘Thank you’ page to help improve overall offer performance.