How content preview impacted the click-through rate on a digest email
NextAfter
Experiment Summary
Timeframe: 08/31/2022 - 09/02/2022
We previously tested adding preview links to a digest email as a strategy to increase click-through rate on a digest email (https://app.winstonknows.com/experiment/4515). In that test, there was a 30% increase in click-throughs on the treatment email.
To further test this, we wanted to go one step further and test using the preview links with our normal content previews beneath vs. just the preview links. We were curious to know if the extra content added value to digest recipients, or if it had no impact on their email engagement.
Research Question
We believe that removing preview content from a digest email for our general subscribers will achieve more click-throughs than a digest email with preview content.
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Some cool new things for you, Vilavanh | 0.88% | ||
| T1: | Some cool new things for you, Danielle | 0.81% | -7.5% | 49.5% |
This experiment has a required sample size of 150,481 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
Results did not validate. It seems removing preview content had no real impact on the click-through rate of the digest email, but results were trending in favor of the control version with preview content.