Does a “following up” subject line increase open rate?
Care Net
Experiment Summary
Timeframe: 03/09/2019 - 03/23/2019
This is an A/B subject line test in our weekly “Weekend Update”
Research Question
This is only the second weekend update that recipients have received from this name. We wanted to do a carry over subject line that referenced the first email sent the week before, and see if it increased opens.
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | A follow up to last week's links | 21.1% | ||
| T1: | How to raise pro abundant life children | 19.9% | -5.7% | 98.2% |
This experiment has a required sample size of 8,684 in order to be valid. Since the experiment had a total sample size of 25,501, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
5.7% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The reference to the email the week before DID increase the open rate.