How changing a headline in an email affects click rate Experiment ID: #7965

National Breast Cancer Foundation

The National Breast Cancer Foundation's mission is to provide help and inspire hope to those affected by breast cancer through early detection, education, and support services.

Experiment Summary

Timeframe: 10/18/2017 - 10/18/2017

National Breast Cancer Foundation was running a high-urgency campaign for Breast Cancer Awareness Month. In email 5, they talked about their free mammography services. They wanted to test whether a more vague headline vs. a headline directly about that content would increase clickthrough rate.

Research Question

Does changing the headline of an email appeal affect clickthrough rate?


C: Control (Question headline)
T1: Topical headline


Treatment Name Click Rate Relative Difference Confidence
C: Control (Question headline) 0.50%
T1: Topical headline 0.43% -13.5% 49.9%

This experiment has a required sample size of 78,366 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

This test did not validate, but the topical headline produced a 13% decrease in clicks.

Experiment Documented by...

Allison Jones

Allison is an Optimization Associate at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.