How email design affects click-through in a digest or newsletter email Experiment ID: #12734


Experiment Summary

Timeframe: 3/14/2019 - 3/18/2019

Email appeal experiments had given us a predisposition to think that email with button, images, and minor design elements are ineffective. But we had really only discovered that to be true in the specific context of a donation appeal.

In this experiment, we wondered if basic design elements could actually lead to greater click-throughs. The control was a plain-text style email in the format we had tested into, and the treatment added section images and clickable html buttons.

Research Question

Will additional design elements in a digest email lead to greater click-through?


C: Plain-Text
T1: Designed


Treatment Name Click Rate Relative Difference Confidence
C: Plain-Text 1.8%
T1: Designed 2.4% 32.0% 92.7%

This experiment has a required sample size of 4,793 in order to be valid. Since the experiment had a total sample size of 8,041, and the level of confidence is not above 95% the experiment results are not valid.

Key Learnings

While this didn’t quite reach our desired level of confidence of 95%, it was very close to validating a 32% increase in clicks. With this type of a result, we’re inclined to test it again to get greater confidence in the learning. There’s the possibility of a novelty effect – it improved clicks simply because it is different, and not necessarily because it is better long term.

Experiment Documented by...

Nathan Hill

Nathan is an Optimization Evangelist at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.