How the subject line of an email affects open rate
Buckner International
Buckner International is a global ministry dedicated to the transformation and restoration of the lives we serve. We are a Christ-centered organization that delivers redemptive ministry to the most vulnerable from the beginning to the ending of life.
Experiment Summary
Timeframe: 11/28/2017 - 11/30/2017
Buckner International sent out an email before Giving Tuesday that let readers know to watch their email for a gift they would be sending them tomorrow. We decided to do a subject line test to see if we could learn more about what motivated Buckner subscribers. For this campaign, we wanted to test whether a mysterious subject line would perform better than one that indicated something was going to be given to them.
Research Question
Does mystery or “generosity” better motivate subscribers?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | It will be given to you | 8.0% | ||
| T1: | Coming tomorrow | 9.0% | 13.3% | 100.0% |
This experiment has a required sample size of 5,306 in order to be valid. Since the experiment had a total sample size of 58,617, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
13.3% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
In this experiment, the subject line that increased a sense of mystery (“Coming tomorrow”) resulted in an increase in opens by 13%. This would seem to indicate that the subscriber base is a bit suspicious of “gifts” being given to them via email. The mystery of “tomorrow” was more motivating to them.