How a specifically defined preview text affects open rate Experiment ID: #8940

NextAfter

Experiment Summary

Timeframe: 4/27/2018 - 5/1/2018

In this experiment, we were sending a weekly update on new research and learnings. We typically let the preview text pull in the opening lines of the email since our emails are always personally written. We wondered if specifically crafting preview text to emphasize the value of the content in the email could increase opens and clicks.

Research Question

Will specific preview text increase opens?

Design

C: Jane, I have a special, spicy, and movie themed update for you
T1: Brent, I have a special, spicy, and movie themed update for you

Results

Treatment Name Open Rate Relative Difference Confidence
C: Jane, I have a special, spicy, and movie themed update for you 40.3%
T1: Brent, I have a special, spicy, and movie themed update for you 36.8% -8.8% 99.8%

This experiment has a required sample size of 1,457 in order to be valid. Since the experiment had a total sample size of 7,237, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    8.8% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

We found that this specifically crafted preview text actually decreased opens by 8.8%. Our hypothesis here is that preview text that sounds and looks like marketing is a dead giveaway that this is a marketing email. We’ve removed an aspect of the personal touch, and this is sets off a flag in the recipients mind that this email is inauthentic.

When looking at clicks, we saw no significant change. So even though we saw a slightly higher click-to-open rate from the more value heavy preview text, the net effect on clicks was a wash.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Nathan Hill

Nathan is an Optimization Evangelist at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.