Which is more effective in a subject line: “Mystery + Value” or “Mystery + Hyper-Personalization”

Experiment ID: #21354


Experiment Summary

Timeframe: 3/4/2020 - 3/18/2020

In this experiment, I was using NextAfter’s 5-step subject line creation tool to craft a subject line for a new webinar. I ended up with 2 different, equally viable subject lines. The first emphasized “utility” or “value” while not giving away the exact topic of the webinar. The second was equally “mysterious” while laying heavily into personalization.


Research Question

Which combination of subject line elements will lead to greater opens?


Treatment Name Open Rate Relative Difference Confidence
C: 21.6%
T1: Matt, I’ve been working on this for you 27.1% 25.5% 100.0%

This experiment has a required sample size of 467 in order to be valid. Since the experiment had a total sample size of 15,818, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    25.5% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The heavily personalized version led to a 25.5% increase in opens at a 99.9% level of confidence. But interestingly enough, there was no statistically significant change in the number of people who signed up for the webinar.

This is yet another example of how the subject line plays a crucial role in setting up how someone will receive and perceive the rest of the email. In this case, it’s likely that people opened the treatment because of the additional personalization and mystery, but ultimately weren’t inclined to join a webinar. While the opens generated by the control were already primed to engage with a webinar – leading to a higher conversion rate from those that did open the email.

In a case like this where the net signups are a wash – I will always err on the side of the email with greater opens. At the most basic level, getting more opens can lead to better deliverability, and lead to stronger relationships with the folks on our email file.

Experiment Documented by Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.

Question about experiment #6488

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.