How a subject line test impacted open rates on a high urgency campaign
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Ended On: 04/01/2021
In an effort to increase overall email engagement during a two-week high urgency annual campaign, we wanted to test various subject lines to measure open and click response. We developed a treatment that used more active rather than passive language. The control subject line asked the question “Are you ready?” while the treatment asked “Are you coming?” while the body copy remained the same for both versions.
Research Question
We believe that creating a new subject line for returning donors will achieve an increase in open rate.
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 24.6% | ||
| T1: | Treatment #1 | 24.8% | 0.7% | 50.2% |
This experiment has a required sample size of 474,054 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
We did not see any valid results from the experiment and only reached 50.2% level of confidence so this concept would need to be re-tested in the future. Looking at the body copy for the treatment, we would want to also test aligning the body copy with the subject line to make sure the original question is addressed or expounded upon to help the message flow more congruently.