How “human” fundraising subject lines affect clickthrough rate
Harvest Ministries
Harvest Christian Fellowship exists to bring Christians closer to God and to bring nonbelievers to a saving relationship with Him by showing how God's Word and faith in Him are applicable and relevant to everyday life.
Experiment Summary
Timeframe: 04/13/2016 - 04/19/2016
Harvest Ministries was sending their mid-month email, which was a highly transactional, premium-focused offer. The offer featured a book that was relevant to Mother’s Day, which was coming up in just a few weeks. The treatment subject line was appropriately transactional: “Order Today to Receive for Mother’s Day”. We had been running some experiments with “humanized” copy — that is, copy that a normal person would write, rather than copy that is explicitly “marketing”. We hypothesized that a more “human” subject line would increase open rate, but we also wanted to measure the effect on clickthrough rate as well, since the two do not operate independently.
At the same time as the subject line test, we were running an email copy and design test as well. The control (which was matched to the control subject line) incentivized the click, a strategy that has shown to attract more clicks, but often less motivated traffic. The treatment made a more direct ask, which often results in fewer clicks but more gifts.
Research Question
Will “humanized copy” increase open rate and increase motivation to click?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Order today | 2.1% | ||
| T1: | Humanized Copy | 0.87% | -58.1% | 100.0% |
This experiment has a required sample size of 766 in order to be valid. Since the experiment had a total sample size of 81,181, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
58.1% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The treatment subject line actually delivered a statistically valid 7.5% increase in open rate. Which is good…if the experiment ended there. But the ultimate goal was gifts, which relies on a click. So we needed to measure the entire funnel.
Once we looked past the subject line, we saw that the treatment email (which made the direct ask) produced a 59% decrease in clickthrough rate, which resulted in a 43% drop in conversion. So, the increase in open rate didn’t really matter that much. It highlights an important learning: open rate is only the beginning of the relationship. Getting 7.5% more people to open the email is a win, but not if it doesn’t pay off when we look at our ultimate metric.
This shows us that “human” subject lines can work, but they may affect reader motivation differently. The control subject line, though it had fewer opens, resulted in many more people showing interest in the offer — most likely due to a combination of the urgency it produced and the design of the control email.