How a unique, more mysterious subject line impacts engagement
NextAfter
Experiment Summary
Ended On: 12/15/2020
For our bi-weekly digest email where we send through a few different links and resources related to online fundraising we wanted to see how we could increase engagement (opens) and also, hopefully, increase the number of clicks. We’ve tested a number of different subject lines like personalization but, generally speaking, we default to something that is very clear as to the utility someone would get from opening the email. So we wanted to see if a pretty unique, creative subject line would stand out more.
In this case, we used a treatment that made it seem like we forgot to create the subject line so it was like a note to ourselves to insert a great subject line. We used some square brackets [like this] to call it out further.
Research Question
We believe that using a very unique, mysterious subject line for email subscribers will achieve a greater open rate..
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 23.8% | ||
| T1: | Treatment #1 | 27.7% | 16.3% | 100.0% |
This experiment has a required sample size of 978 in order to be valid. Since the experiment had a total sample size of 14,079, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
16.3% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
With almost a 100% LoC (99.9%) the unique, mystery subject line led to an increase in opens which more than anything underscores how changing up the type of subject lines used can help keep people engaged. In this case, the [square brackets] may have helped it stand out but our theory is the more humourous and unique element of the subject line helped lead to more opens.
Interestingly, with a true 100% LoC (100%) we saw a 93.3% increase in click rate and a 63.8% increase in click to open rate (99.9% LoC) with the unique, mysterious subject line. This underscores both the relationship between the open rate and click rate and how the subject line can influence a reader’s motivation before they even get into the body/content of the email itself.