How subject line urgency impacted email performance
Illinois Policy Institute
Experiment Summary
Timeframe: 12/30/2016 - 01/13/2017
As the year end deadline approached for Illinois Policy Institute’s year end fundraising campaign, they wanted to find ways to communicate that urgency to the subscribers. For the second-to-last email send, sent out on December 30th, they decided to run a subject line test to determine which would have the biggest impact on subscriber engagement.
Both of the subject lines communicated urgency. The primary difference between the two was that one, time is running out, is more direct in its approach and is a more common approach taken by many organizations in their fundraising and marketing-style emails. The alternative, tick tock, was a bit less common and did not give away the purpose of the email.
Research Question
Which subject line will have the biggest impact on subscriber engagement (opens and clicks)?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | tick tock | 18.5% | ||
| T1: | time is running out | 17.1% | -7.7% | 100.0% |
This experiment has a required sample size of 5,457 in order to be valid. Since the experiment had a total sample size of 61,276, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
7.7% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
We found that the more direct and “sale-y” subject line of time is running out decreased the open rate by 8%. Additionally, this lower open rate led to an incrementally lower click through rate with a 33% decrease (with 94.6% statistical validity). Our hypothesis is that with the time is running out subject line, the subscribers immediately felt marketed to since it is not a subject line most people would use in their day to day communications. As a result, they were less likely to engage.
This is a great example of the concept that people give to people. As fundraisers, we want to communicate urgency but we need to do it in such a way that it still feels genuine.