The impact of an unusual, mysterious, personally direct subject line on open rates
Care Net
Experiment Summary
Ended On: 05/14/2019
We tested this in our weekly “weekend update” email. We are always eager to improve email open rates.
Research Question
We wanted to know if telling the email recipients in the subject line that we we had specifically recorded something for them would increase open rates. This combined mystery, personal pronouns, and unusual phrasing compared to what we usually send to them.
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | What real pro-life victory looks like | 11.2% | ||
| T1: | We recorded this for you | 12.7% | 13.8% | 100.0% |
This experiment has a required sample size of 3,425 in order to be valid. Since the experiment had a total sample size of 81,227, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
13.8% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The result was a definitely lift in open rates, but we suspect that this kind of subject line shouldn’t be overused or it will lose its power and become gimmicky.