How increased specificity in a subject line affects open rate

National Breast Cancer Foundation

The National Breast Cancer Foundation's mission is to provide help and inspire hope to those affected by breast cancer through early detection, education, and support services.

Experiment Summary

Timeframe: 09/30/2016 - 10/03/2016

The National Breast Cancer Foundation sent an offer to a list of people who had signed up for their Early Detection Plan app. The offer was a free eBook called What Every Woman Needs To Know that included resources related to early detection of breast cancer.

They decided to test subject lines on this email. The control subject line teased the outcome of the eBook: “Stay informed about your breast health”. They hypothesized that being extremely clear about the offer, even in the subject line, might increase open rate since the list was so highly qualified. So they developed a treatment subject line: “A new early detection guide”.

They launched an A/B test to determine a winner.

Research Question

Will increasing specificity increase open rate?

Design

C: Desired Effect

T1: Specificity

Results

This experiment has a required sample size of 3,854 in order to be valid. Since the experiment had a total sample size of 12,352, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    15.8% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

Though both emails had a relatively low open rate, the treatment subject line produced a 15.8% increase over the control. This shows that increased clarity can increased the perceived value of the email and incentivize readers to open.

“Desired effect” language has proven powerful before, but in this case, clarity trumped persuasion.