How a more humanized subject line affects open rate on an email
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Ended On: 04/05/2021
In an effort to increase overall email engagement during a two-week high urgency annual campaign, we wanted to test various subject lines to measure open and click response. We wanted to test the effect of a more humanized subject line vs. a standard subject line in an email. The control subject line read “Membership question” and the treatment read “I hope you will be joining us.”
Research Question
We believe that using a humanized subject line for a housefile audience will achieve an increase in open rates .
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 24.5% | ||
| T1: | Treatment #1 | 24.7% | 0.7% | 49.3% |
This experiment has a required sample size of 503,274 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
We saw a .69% increase in traffic on the treatment but this test did not validate because it resulted in a confidence level of 49.2%. This would require retesting and tested with a prospecting email instead of a housefile email in order to test a larger sample size. We also recommend testing a more humanized subject line again using a first name in the subject line instead of “you” if testing the housefile email again.