How email design affects click-through in a digest or newsletter email Experiment ID: #12734
Timeframe: 3/14/2019 - 3/18/2019
Email appeal experiments had given us a predisposition to think that email with button, images, and minor design elements are ineffective. But we had really only discovered that to be true in the specific context of a donation appeal.
In this experiment, we wondered if basic design elements could actually lead to greater click-throughs. The control was a plain-text style email in the format we had tested into, and the treatment added section images and clickable html buttons.
Will additional design elements in a digest email lead to greater click-through?
|Treatment Name||Click Rate||Relative Difference||Confidence|
This experiment has a required sample size of 4,793 in order to be valid. Since the experiment had a total sample size of 8,041, and the level of confidence is not above 95% the experiment results are not valid.
While this didn’t quite reach our desired level of confidence of 95%, it was very close to validating a 32% increase in clicks. With this type of a result, we’re inclined to test it again to get greater confidence in the learning. There’s the possibility of a novelty effect – it improved clicks simply because it is different, and not necessarily because it is better long term.