How the tone of a subject line impacts open rate
Live Action
Experiment Summary
Timeframe: 01/24/2017 - 02/01/2017
Live Action produces investigative videos where they go undercover inside abortion facilities. These investigative videos receive millions of views. Whenever they release a new investigative video, they send it out to their email house file first. Traditionally, the subject line hints at the release of the latest video by saying “NEW VIDEO: Prenatal care nonexistent at Planned Parenthood”. They wandered if the would be able to increase the number of people who opened the email if they took a more casual approach and didn’t mention that the email was a “new video” release. They developed a treatment with the subject line: “One option: abortion:”. They A/B tested the two different subject lines.
Research Question
Which subject line would get a higher open rate?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | NEW VIDEO: Prenatal care nonexistent at Planned Parenthood | 9.8% | ||
| T1: | One option: abortion | 11.3% | 14.2% | 100.0% |
This experiment has a required sample size of 3,684 in order to be valid. Since the experiment had a total sample size of 333,889, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
14.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The treatment email that didn’t mention the new video in the subject line had a 14% increase in opens. While it might seem strategic to mention the new video in the subject line since that is what the email is about, the tone of the control subject line may have come off too organization-centric causing less people to want to open it. When subject lines are more casual and believable that Lila has actually sent the email, people are more motivated to open them.