How a specifically defined preview text affects open rate
NextAfter
Experiment Summary
Timeframe: 04/27/2018 - 05/01/2018
In this experiment, we were sending a weekly update on new research and learnings. We typically let the preview text pull in the opening lines of the email since our emails are always personally written. We wondered if specifically crafting preview text to emphasize the value of the content in the email could increase opens and clicks.
Research Question
Will specific preview text increase opens?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Jane, I have a special, spicy, and movie themed update for you | 40.3% | ||
| T1: | Brent, I have a special, spicy, and movie themed update for you | 36.8% | -8.8% | 99.8% |
This experiment has a required sample size of 1,457 in order to be valid. Since the experiment had a total sample size of 7,237, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
8.8% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
We found that this specifically crafted preview text actually decreased opens by 8.8%. Our hypothesis here is that preview text that sounds and looks like marketing is a dead giveaway that this is a marketing email. We’ve removed an aspect of the personal touch, and this is sets off a flag in the recipients mind that this email is inauthentic.
When looking at clicks, we saw no significant change. So even though we saw a slightly higher click-to-open rate from the more value heavy preview text, the net effect on clicks was a wash.