How email preheaders affect open rate
Harvest Ministries
Harvest Christian Fellowship exists to bring Christians closer to God and to bring nonbelievers to a saving relationship with Him by showing how God's Word and faith in Him are applicable and relevant to everyday life.
Experiment Summary
Timeframe: 05/30/2016 - 06/06/2016
Harvest Ministries was scheduled to send their mid-month fundraising update email. As their team was setting up the email, they developed a hypothesis that the preheader copy (or preview copy designed to be seen before the email is opened) might be decreasing open rate. Though this copy is strongly encouraged by marketers and email service providers, it obscured the personalized greeting and didn’t make sense when read left to right.
They hypothesized that removing this could increase the appearance that this was a personal email, not a marketing email. They split their file and launched a test to find out how employing a preheader affects open rate.
Research Question
Will removing the preheader increase open rate?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Preheader | 24.7% | ||
| T1: | No Preheader | 25.2% | 2.2% | 99.6% |
This experiment has a required sample size of 46,826 in order to be valid. Since the experiment had a total sample size of 206,889, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
2.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The treatment, which did not use a preheader, increased open rate by 2.2%. While this may not seem like a significant amount, it certainly matters at scale. Harvest had around 220,000 people included in this send, so that’s an additional 4,400 recipients who would have opened the email — which means more clicks, donors, and donations.
This prompted another round of testing to understand whether the personalization elements were specifically responsible for increasing open rate.