How a more shocking subject line impacts clicks
Leadership Institute
Experiment Summary
Timeframe: 08/15/2017 - 08/29/2017
The Leadership Institute sent an email out about president Morton Blackwell’s interview with the BBC. During the interview, Morton reminisced on President Ronald Reagan’s anti-communism jokes. The Leadership Institute decided to test a subject line to see if whether mentioning a more shocking line (“Reagan’s bombing joke”) would generate more clicks than a more politically correct, “safe” subject line (Timely BBC interview about President Reagan).
Research Question
Will a less politically correct and “shocking” subject line get more opens than a Morton-esque subject line?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Timely BBC interview about President Reagan | 2.3% | ||
| T1: | Reagan's bombing joke | 4.9% | 112.5% | 99.1% |
This experiment has a required sample size of 395 in order to be valid. Since the experiment had a total sample size of 1,382, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
112.5% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The more shocking subject line generated 34 clicks, rather than the 16 clicks of the more politically correct subject line. This 112.5% increase in traffic shows that the Leadership Institute can excite and engage their readers with a more shocking subject line.
The open rates for each of the subject lines are as follows: 866 opens at an 18.2% rate for the control and 1,133 opens at a 23.8% rate for the treatment. This is a 30.8% lift in opens.