Impact of a direct subject line vs. a vague one
Care Net
Experiment Summary
Timeframe: 06/08/2019 - 06/11/2019
Every Saturday Care Net sends out a “weekend update” email highlighting 1-2 of our blog posts. During some variation testing, vague but interesting subject lines had been performing well, but I was not convinced they would win every time. This week I decided to test it against a direct subject line that I knew was quite strong.
Research Question
Do mysterious, vague, subject lines win all the time?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | An encouraging headline! | 33.0% | ||
| T1: | Pastor calls cohabiting couples to marry | 38.8% | 17.5% | 100.0% |
This experiment has a required sample size of 531 in order to be valid. Since the experiment had a total sample size of 32,113, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
17.5% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
My suspicions were right, the stronger but direct subject line did much better. I do want to test it again with a more “direct but boring” subject line.