How a relevant subject line not only impacts opens, but clicks and donations
Harvest Ministries
Harvest Christian Fellowship exists to bring Christians closer to God and to bring nonbelievers to a saving relationship with Him by showing how God's Word and faith in Him are applicable and relevant to everyday life.
Experiment Summary
Timeframe: 06/09/2017 - 06/20/2017
Harvest Ministries had a high-urgency fundraising campaign leading up to a large-scale evangelism event, Harvest America. They sent a series of emails during this campaign to acquire donations and revenue. In an effort to increase the number of people who opened their emails and ultimately give, they tested a subject line of an email as they got closer to the campaign’s final day. The control used the subject line, “One Thing Remains” and the treatment tested something that was more timely, “This Sunday…”
Research Question
Which subject line would get more people to open the email?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | One Thing Remains | 21.3% | ||
| T1: | This Sunday... | 22.6% | 5.8% | 100.0% |
This experiment has a required sample size of 8,698 in order to be valid. Since the experiment had a total sample size of 316,376, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
5.8% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
While both subject lines were relatively vague, the treatment subject line increased opens by 5.8%. The treatment was more relevant to the campaign and created a sense of urgency. Because we were able to get almost 6% more people to open the email, the relevancy of the subject line with the email copy also lead to a 9% increase in clicks and ultimately 28% more donations!