Will a more clear and direct subject line lead to more course signups?
NextAfter
Experiment Summary
Ended On: 05/21/2021
We wanted to let people know about an updated online course that was now available for free. We’ve seen how more mysterious subject lines can evoke a sense of curiosity and get more opens but we’ve also seen that a clear direct subject line can sometimes lead to more action. So we put it to the test. The control was a more mysterious subject line that said there was ‘exciting news for you’ but didn’t say anything about what it was while the treatment was clear with the fact that there was a ‘new, free course for you’. Both subject lines used personalization in the first name in the subject line.
Research Question
We believe that being more clear with the offer in the email in the subject line for email subscribers will achieve more email course signups.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.74% | ||
| T1: | Treatment #1 | 0.89% | 19.8% | 68.9% |
This experiment has a required sample size of 28,740 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
While the final result on conversion wasn’t statistically significant, there was a directional lift towards the Treatment increasing conversion. If we go ‘up the funnel’ or ‘down the mountain’ and look at clicks, we actually observed 15.8% increase in clicks with a 91% LoC which does indicate more intent to sign up from the clean subject line compared to the mysterious subject line. And when we also see that open rate for the clear subject line was 11% lower with 100% LoC, it does seem to validate both the hypothesis that:
- Mysterious subject lines are better at generating opens
- Clear subject lines are better at generating actions
Moving forward, while the mysterious subject line strategy could work for driving action — maybe the increase in opens can overcome the decrease in clicks and still lead to more actions/conversions — it is probably best used for driving traffic to blog posts or getting reply backs or other engagement beyond conversions. And for conversion-focused emails being more simple, clear, and direct seems to be a better approach.
Worth testing would be a combination or adding a bit more mystery to the clear and direct course subject line.