How a more conversational tone in a subject line impacts open rate
Focus on the Family
Focus on the Family is a global Christian ministry dedicated to helping families thrive. We provide help and resources for couples to build healthy marriages that reflect God's design, and for parents to raise their children according to morals and values grounded in biblical principles.
Experiment Summary
Timeframe: 12/12/2017 - 12/19/2017
The emails for Focus on the Family’s year-end giving campaign are image heavy with a video on them to push people to the donation page. We hypothesized that a personal email, with a more conversational tone from the subject line to the body copy would increase conversion. The first goal was to get more people to open the emails in an effort to get more people to click and convert. We A/B split the email file and tested it.
Research Question
Which email would get more people to open it?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | You still have time to double your... | 16.7% | ||
| T1: | Nothing is impossible for God | 29.6% | 77.1% | 100.0% |
This experiment has a required sample size of 82 in order to be valid. Since the experiment had a total sample size of 205,068, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
77.1% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
While we weren’t able to validate this experiment based off of donations, which is the ultimate goal, we were able to validate the experiment on opens. The treatment subject line took a more conversational approach that didn’t give any reference to giving or donating by saying, “Nothing is impossible for God”. This approach increased opens by 77%! Knowing that we can significantly increase the amount of people who see and read the email with this kind of approach, we recommend testing this approach again and doing a 50/50 split of the email file so that we have a greater chance at validating it off of donations to see what the impact of this approach really has on donor conversion.