How a specifically defined preview text affects open rate - NextAfter
NextAfter

How a specifically defined preview text affects open rate

Experiment ID: #5952

NextAfter

Experiment Summary

Timeframe: 04/27/2018 - 05/01/2018

In this experiment, we were sending a weekly update on new research and learnings. We typically let the preview text pull in the opening lines of the email since our emails are always personally written. We wondered if specifically crafting preview text to emphasize the value of the content in the email could increase opens and clicks.

Research Question

Will specific preview text increase opens?

Design

C: Jane, I have a special, spicy, and movie themed update for you
T1: Brent, I have a special, spicy, and movie themed update for you

Results

  Treatment Name Open Rate Relative Difference Confidence
C: Jane, I have a special, spicy, and movie themed update for you 40.3%
T1: Brent, I have a special, spicy, and movie themed update for you 36.8% -8.8% 99.8%

This experiment has a required sample size of 1,457 in order to be valid. Since the experiment had a total sample size of 7,237, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    8.8% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

We found that this specifically crafted preview text actually decreased opens by 8.8%. Our hypothesis here is that preview text that sounds and looks like marketing is a dead giveaway that this is a marketing email. We’ve removed an aspect of the personal touch, and this is sets off a flag in the recipients mind that this email is inauthentic.

When looking at clicks, we saw no significant change. So even though we saw a slightly higher click-to-open rate from the more value heavy preview text, the net effect on clicks was a wash.


Experiment Documented by Nathan Hill
Nathan Hill is Vice President of Marketing at NextAfter.

Question about experiment #5952

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.