How reducing specificity affects email acquisition Experiment ID: #2697

Hoover Institution

Experiment Summary

Timeframe: 10/1/2015 - 11/24/2015

The Hoover Institution launched a campaign to grow the subscriber base of the Hoover Daily Report, their flagship policy newsletter. They drafted landing page copy that highlighted the authority of their research fellows (who provide content for the Daily Report), but wanted to test a much simpler value proposition inspired by another email newsletter. They hypothesized that this new simple headline would increase email acquisition rate.

Research Question

Will a simpler headline clarify the value proposition and increase email acquisition rate?

Design

C: Authority
T1: Simplified Headline

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Authority 60.5%
T1: Simplified Headline 64.3% 6.2% 92.7%

This experiment has a required sample size of 1,290 in order to be valid. Since the experiment had a total sample size of 2,161, and the level of confidence is not above 95% the experiment results are not valid.

Key Learnings

Though it was only a slight lift, there was enough difference to achieve statistical validity. The simpler headline produced a 6% lift in email conversion — which tells us that we don’t always have to be selling. Many times, our prospects can be attracted to subscribe by clarity rather than perceived authority.


Experiment Documented by...

Jeff Giddens

Jeff is the President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.