How clarity in the subject line impacts click-through
Live Action
Experiment Summary
Timeframe: 02/14/2017 - 02/27/2017
For the release of a new investigative video, Live Action sent an email to their house file and included a link for people to view it. In an effort to get more people to open and click through to watch the video, they tested two different subject lines. The control took a personal approach by including the recipients name in the subject line. The treatment was much more matter of fact and told people exactly what the email was about. They split their file and tested.
Research Question
Which subject line would get more people to click-through?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | *|MERGE16|*, it’s up to you | 0.73% | ||
| T1: | Former managers: Planned Parenthood is not health care | 0.88% | 20.9% | 100.0% |
This experiment has a required sample size of 26,396 in order to be valid. Since the experiment had a total sample size of 382,803, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
20.9% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
Their wasn’t a significant difference in open rate between the control and the treatment, but the treatment got a 20% increase in click-through. What we can learn from this is that:
- Clarity trumps persuasion. While the control subject line was personalized and this can be a highly motivating tactic, it was also very unclear and may have caused some friction. The treatment told people exactly the topic of the email.
- The treatment subject line helped reinforce the email value proposition and strengthened the message, resulting in more people clicking-through from the treatment.