Which is more effective in a subject line: “Mystery + Value” or “Mystery + Hyper-Personalization”
NextAfter
Experiment Summary
Timeframe: 03/04/2020 - 03/18/2020
In this experiment, I was using NextAfter’s 5-step subject line creation tool to craft a subject line for a new webinar. I ended up with 2 different, equally viable subject lines. The first emphasized “utility” or “value” while not giving away the exact topic of the webinar. The second was equally “mysterious” while laying heavily into personalization.
Research Question
Which combination of subject line elements will lead to greater opens?
Design
Results
Treatment Name | Open Rate | Relative Difference | Confidence | |
---|---|---|---|---|
C: | A new fundraising webinar you’ll want to mark your calendar for | 21.6% | ||
T1: | Matt, I’ve been working on this for you | 27.1% | 25.5% | 100.0% |
This experiment has a required sample size of 467 in order to be valid. Since the experiment had a total sample size of 15,818, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
25.5% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The heavily personalized version led to a 25.5% increase in opens at a 99.9% level of confidence. But interestingly enough, there was no statistically significant change in the number of people who signed up for the webinar.
This is yet another example of how the subject line plays a crucial role in setting up how someone will receive and perceive the rest of the email. In this case, it’s likely that people opened the treatment because of the additional personalization and mystery, but ultimately weren’t inclined to join a webinar. While the opens generated by the control were already primed to engage with a webinar – leading to a higher conversion rate from those that did open the email.
In a case like this where the net signups are a wash – I will always err on the side of the email with greater opens. At the most basic level, getting more opens can lead to better deliverability, and lead to stronger relationships with the folks on our email file.
Question about experiment #28789
If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.