How emotional appeal versus specific outcome affect open rates
Bill of Rights Institute
Experiment Summary
Timeframe: 08/01/2020 - 08/07/2020
Historically, much of BRI’s subject lines have focused on stating the topic of the email without adding emotional context. From experiments with similar organizations, we have found that using emotional language can help prompt the action that we’re striving for (such as opening or clicking on the email.) We wanted to test this same concept for BRI’s audience.
Research Question
Will emotional language in the subject line prompt more engagement when compared to one that is purely outcome-oriented?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Advancing logical and civil discourse | 17.6% | ||
| T1: | Don’t lose hope – together we’re making a difference | 18.7% | 6.1% | 88.4% |
This experiment has a required sample size of 9,775 in order to be valid. Since the experiment had a total sample size of 12,580, and the level of confidence is not above 95% the experiment results are not valid.
Key Learnings
In this situation, we did not see any statistically significant change between the two subject lines. As with most subject line experiments, this same concept will need to be tested over a series of emails to better understand the impact of this hypothesis for the BRI audience.