How increasing relevancy of the subject line affects open rate
National Breast Cancer Foundation
The National Breast Cancer Foundation's mission is to provide help and inspire hope to those affected by breast cancer through early detection, education, and support services.
Experiment Summary
Timeframe: 10/01/2016 - 10/13/2016
National Breast Cancer Foundation send out a monthly newsletter to keep donors and supporters informed and engaged. They had traditionally used a functional subject line that clearly stated what the communication was—in this case, “October 2016 Newsletter | Helping Women Now”.
But they realized that the subject line is the initial point of interaction with their readers—and if they didn’t make it “open-worthy”, they might be causing some possible readers to delete it outright.
So they drafted a second subject line to give more insight into the communications and provide a greater incentive to open: “It’s A Special Month…”.
They split their email file to determine a winner.
Research Question
How will increasing the context of the subject line affect open rate?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Standard Subject Line | 15.5% | ||
| T1: | Increased relevancy | 17.1% | 10.4% | 95.9% |
This experiment has a required sample size of 4,050 in order to be valid. Since the experiment had a total sample size of 8,809, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
10.4% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The new subject line produced a 10.4% increase in opens, which meant that they were able to engage a larger segment of their email file. This experiment also gave insight to future subject line testing by showing that increasing the force of the incentive to open the email produced a tangible increase in engagement.