How subject line testing affects clickthrough
Illinois Policy Institute
Experiment Summary
Timeframe: 01/24/2015 - 01/26/2015
As part of their monthly appeal strategy, Hillsdale College ran an experiment to a segment of their house email list to determine the best messaging approach for the appeal. A clear winner was identified and the resulting email was scheduled to be sent out to the remainder of the email list with a subject line A/B split test.
Research Question
How does a subject line that is more aspirational and positive affect opens and clicks to an email?.
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Hillsdale's plan to reverse damage | 4.5% | ||
| T1: | A message of hope | 2.9% | -35.5% | 100.0% |
This experiment has a required sample size of 1,079 in order to be valid. Since the experiment had a total sample size of 305,409, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
35.5% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The “Message of hope” ended up with 10.1% less opens and 35.5% less clicks. The specificity of the control subject line, “Hillsdale’s plan to reverse damage,” proved to be more effective in inspiring recipients to open the email and then click through to the landing page. It is important to remember that specificity yields clarity, and that clarity trumps persuasion.