How the type of subject line affects open rate in an appeal Experiment ID: #8483

National Breast Cancer Foundation

The National Breast Cancer Foundation's mission is to provide help and inspire hope to those affected by breast cancer through early detection, education, and support services.

Experiment Summary

Ended On: 1/24/2018

NBCF was sending out an email appeal to their housefile for their Valentine’s Day Hope Kit campaign. Their goal for the experiment was to get more eyes on the email and more opens. The hypothesis was that the subject line “720 women” would be more compelling than asking the reader a question or to do something, for example, “Will you help me do this?”. The email itself remained the same

Research Question

Does the type of subject in an email affect open rate?


Treatment Name Open Rate Relative Difference Confidence
C: Will you help me share some love? 13.7%
T1: 720 women 16.3% 19.2% 100.0%

This experiment has a required sample size of 1,418 in order to be valid. Since the experiment had a total sample size of 23,070, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    19.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The treatment headline produced a 19% increase in open rate of the email appeal. This is possibly due to the workload involved in the first subject line, which requires “work” on the reader’s behalf and immediately adds friction to the opening process, and partially due to the compelling nature of the treatment, “720 women”, which refers to the 720 women diagnosed with breast cancer every day.

Experiment Documented by...

Allison Jones

Allison is an Optimization Associate at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.