Will a more clear and direct subject line lead to more course signups? | NextAfter
NextAfter

Will a more clear and direct subject line lead to more course signups?

Experiment ID: #61800

NextAfter

Experiment Summary

Ended On: 05/21/2021

We wanted to let people know about an updated online course that was now available for free. We’ve seen how more mysterious subject lines can evoke a sense of curiosity and get more opens but we’ve also seen that a clear direct subject line can sometimes lead to more action. So we put it to the test. The control was a more mysterious subject line that said there was ‘exciting news for you’ but didn’t say anything about what it was while the treatment was clear with the fact that there was a ‘new, free course for you’. Both subject lines used personalization in the first name in the subject line.

Research Question

We believe that being more clear with the offer in the email in the subject line for email subscribers will achieve more email course signups.

Design

C: Control
T1: Treatment #1

Results

  Treatment Name Conv. Rate Relative Difference Confidence
C: Control 0.88%
T1: Treatment #1 0.74% -15.8% 66.5%

This experiment has a required sample size of 31,786 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

While the final result on conversion wasn’t statistically significant, there was a directional lift towards the Treatment increasing conversion. If we go ‘up the funnel’ or ‘down the mountain’ and look at clicks, we actually observed 15.8% increase in clicks with a 91% LoC which does indicate more intent to sign up from the clean subject line compared to the mysterious subject line. And when we also see that open rate for the clear subject line was 11% lower with 100% LoC, it does seem to validate both the hypothesis that:

  1. Mysterious subject lines are better at generating opens
  2. Clear subject lines are better at generating actions

Moving forward, while the mysterious subject line strategy could work for driving action — maybe the increase in opens can overcome the decrease in clicks and still lead to more actions/conversions — it is probably best used for driving traffic to blog posts or getting reply backs or other engagement beyond conversions. And for conversion-focused emails being more simple, clear, and direct seems to be a better approach.

Worth testing would be a combination or adding a bit more mystery to the clear and direct course subject line.


Experiment Documented by NextAfter

Question about experiment #61800

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.