How a detailed subject line performs against a less detailed subject line
Leadership Institute
Experiment Summary
Timeframe: 05/30/2016 - 06/13/2016
The Leadership Institute sent out a Memorial Day email, and wanted to know if using subject lines that talked about or alluded to Memorial Day would perform better than a brief subject line that doesn’t let the reader know what the content is about.
Research Question
Do people want to know what’s in the email before they open?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Remembering on this Memorial Day | 14.4% | ||
| T1: | One moment | 18.1% | 26.2% | 100.0% |
| T2: | Training and sacrifice | 16.9% | 17.4% | 100.0% |
This experiment has a required sample size of 1,115 in order to be valid. Since the experiment had a total sample size of 32,784, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
26.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
While both treatments improved the open rate, the first treatment did much better, with an 18.1% open rate compared to the 16.9% open rate in the second treatment subject line of “Training and sacrifice” and a 14.4% open rate with the control of “Remembering on this Memorial Day”. This resulted in a 26.2% increase in traffic. The “One moment” subject line may resonate with the readers because it indicates that it should not take long to read the email. In the future, indicating that reading the email will not take too much time should be further tested against other successful subject lines to determine its efficacy in relation to those subject lines. Control had 27 clicks and a .25% click rate; Treatment 1 had 16 clicks and a .15% click rate; Treatment 2 had 9 clicks and a .08% click rate.