How urgency in the subject line impacts open and click rates
CRISTA Ministries
Experiment Summary
Timeframe: 01/21/2017 - 12/30/2016
World Concern sent a series of 7 fundraising emails as part of their year end campaign. They wanted to ensure each email was getting the maximum impact so they set up different treatments for each email.
For the second email in the series, we wanted to discover if adding urgency to the subject line would impact the percentage of people that either opened or clicked on the message. We decided to test a more vague subject line against one that conveyed the urgency of the December 31st deadline since that was the topic of the email.
Research Question
Will adding urgency to the subject line increase the percentage of people the open or click on the message?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Today | 18.9% | ||
| T1: | Don’t wait another minute | 13.7% | -27.5% | 100.0% |
This experiment has a required sample size of 386 in order to be valid. Since the experiment had a total sample size of 10,043, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
27.5% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
In the end, the more urgent subject line of “Don’t wait another minute” decreased the open rate of the email by 27.5%. However, we also wanted to monitor how it impacted the click through rate since the ultimate goal of this email was to get people to the donation page. When analyzed, we discovered there was not a significant difference between the percentage of people clicking on a link.
As a rule of thumb, it is a good trade off to get less opens if it means more people clicking on the link but with there being no discernible difference between click rates, we would rather have more people reading the email so they are exposed to the message when emails later in the series are sent.