How email preheaders affect open rate - NextAfter
Harvest Ministries

How email preheaders affect open rate

Experiment ID: #4221

Harvest Ministries

Harvest Christian Fellowship exists to bring Christians closer to God and to bring nonbelievers to a saving relationship with Him by showing how God's Word and faith in Him are applicable and relevant to everyday life.

Experiment Summary

Timeframe: 05/30/2016 - 06/06/2016

Harvest Ministries was scheduled to send their mid-month fundraising update email. As their team was setting up the email, they developed a hypothesis that the preheader copy (or preview copy designed to be seen before the email is opened) might be decreasing open rate. Though this copy is strongly encouraged by marketers and email service providers, it obscured the personalized greeting and didn’t make sense when read left to right.

They hypothesized that removing this could increase the appearance that this was a personal email, not a marketing email. They split their file and launched a test to find out how employing a preheader affects open rate.

Research Question

Will removing the preheader increase open rate?

Design

C: Preheader
T1: No Preheader

Results

  Treatment Name Open Rate Relative Difference Confidence
C: Preheader 24.7%
T1: No Preheader 25.2% 2.2% 99.6%

This experiment has a required sample size of 46,826 in order to be valid. Since the experiment had a total sample size of 206,889, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    2.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The treatment, which did not use a preheader, increased open rate by 2.2%. While this may not seem like a significant amount, it certainly matters at scale. Harvest had around 220,000 people included in this send, so that’s an additional 4,400 recipients who would have opened the email — which means more clicks, donors, and donations.

This prompted another round of testing to understand whether the personalization elements were specifically responsible for increasing open rate.


Experiment Documented by Jeff Giddens
Jeff Giddens is President of NextAfter.

Question about experiment #4221

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.