How including a hot button issue in the subject line affects open rate
Leadership Institute
Experiment Summary
Timeframe: 07/13/2018 - 07/14/2018
The Leadership Institute works with college students to help defend their free speech on campus. They wanted to see if including hot button terms like “free speech” and “the one percent” in the subject line would affect open rates. Their hypothesis was that the subject line about free speech would generate more opens.
Research Question
Will mentioning free speech in the subject line improve the open rate of the email?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | The one percent at UM-Amherst | 20.4% | ||
| T1: | Free speech at UM-Amherst | 19.9% | -2.6% | 64.4% |
This experiment has a required sample size of 42,595 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
The open rates for each email were no more than .4% deviations away from 20%. This experiment was not conclusive. It’s likely that the hot button issues did not affect open rate as much due to the specified name of the school in the subject line. A further experiment without a specified name may render desired results to confidently conclude this experiment.