How email design affects click-through in a digest or newsletter email
NextAfter
Experiment Summary
Timeframe: 03/14/2019 - 03/18/2019
Email appeal experiments had given us a predisposition to think that email with button, images, and minor design elements are ineffective. But we had really only discovered that to be true in the specific context of a donation appeal.
In this experiment, we wondered if basic design elements could actually lead to greater click-throughs. The control was a plain-text style email in the format we had tested into, and the treatment added section images and clickable html buttons.
Research Question
Will additional design elements in a digest email lead to greater click-through?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Plain-Text | 1.8% | ||
| T1: | Designed | 2.4% | 32.0% | 92.7% |
This experiment has a required sample size of 4,793 in order to be valid. Since the experiment had a total sample size of 8,041, and the level of confidence is not above 95% the experiment results are not valid.
Key Learnings
While this didn’t quite reach our desired level of confidence of 95%, it was very close to validating a 32% increase in clicks. With this type of a result, we’re inclined to test it again to get greater confidence in the learning. There’s the possibility of a novelty effect – it improved clicks simply because it is different, and not necessarily because it is better long term.