How more mystery in a subject line affects click-rate
NextAfter
Experiment Summary
Timeframe: 06/27/2019 - 07/22/2019
In this webinar follow-up email, we had been using the same kind of subject line for a while. While the sample size is low and hard to validate on, we wanted to put a hypothesis to the test that a subject line with more mystery would lead to more opens and clicks.
Research Question
Will a subject line with more mystery lead to more opens and clicks?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | The webinar recording is ready for you | 25.1% | ||
| T1: | It's ready for you! | 27.8% | 11.0% | 60.8% |
This experiment has a required sample size of 1,971 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
As we assumed, our sample size was not large enough to validate the result, despite seeing an 11% increase in clicks from the treatment version. A more radical test will be required in order to see a statistically confidence result.