How a softer subject line impacts open rate
Ronald Reagan Presidential Foundation & Institute
Experiment Summary
Timeframe: 05/30/2017 - 06/13/2017
The Reagan Presidential Library was promoting their summer events through a promotional email series. They had historically used the same subject line “Make Your Reservations for Our Upcoming Events at the Reagan Library.”
They decided to test out a new approach to this subject line by using one that did not use an action the reader had to take in the test: “Announcing a New Event at the Reagan Library!”
Research Question
Which subject line will get the most subscribers to engage?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Make Your Reservations for Our Upcomi... | 17.2% | ||
| T1: | Announcing a New Event at the Reagan... | 18.8% | 8.9% | 100.0% |
This experiment has a required sample size of 4,832 in order to be valid. Since the experiment had a total sample size of 129,744, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
8.9% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The treatment resulted in a 8.9% increase in overall email opens. This increased lift in opens led to an 8.3% increase in the number of visitors (clicks) to the website.
Since the control asks the reader to “Make their reservation,” those not willing to make that commitment were less inclined to open it when they received it.
By eliminating the immediate request for action and focusing the subject line solely on what someone get’s (or get’s to discover, in this case), utilizing mystery, we convinced approximately 9% more readers to open.
Had there not been any mystery in the original subject line, it is possible that the difference in performance would be substantially greater.