How creating intrigue in an email promoting new research impacted email signup rates
NextAfter
Experiment Summary
Ended On: 09/08/2022
As a part of launching our new research study for how the industry handles a New Donor Welcome Series, we decided to experiment with a bullet list of questions that the research would answer, thereby increasing the “intrigue” factor for the reader.
Our thought was that this intrigue would drive more motivated clicks and increase email signup rates.
Research Question
We believe that building intrigue for subscribers we are promoting new research to will achieve increase email signup rates.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.80% | ||
| T1: | Treatment #1 | 0.74% | -7.6% | 48.3% |
This experiment has a required sample size of 160,177 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
Unfortunately, this treatment did not have any substantial impact on the email signup rate. Although it shows a -7.6% email signup rate, the level of confidence is less than a coin flip probability to be repeated.
This means that the treatment did not create more (or less) motivation for a subscriber to get our new research.
This could simply be due to the fact that new research is just in such high demand that any lower-level experiments to manufacture “more demand” aren’t necessary. On the other hand, it could be due to the fact that this approach was ineffective and should be retested in future opportunities to re-test the concept.