How removing donation language from the CTA of an email appeal impacts click-through and donations.
Harvest Ministries
Harvest Christian Fellowship exists to bring Christians closer to God and to bring nonbelievers to a saving relationship with Him by showing how God's Word and faith in Him are applicable and relevant to everyday life.
Experiment Summary
Timeframe: 01/18/2017 - 01/23/2017
Harvest Ministries monthly appeal message is always focused on the content of the premium they are giving that month. The premium is a gift to the subscriber if they give a donation. Because the message of the email is centered around the content of the premium, they wondered if they would be able to get more clicks and ultimately more donations if they made this the primary CTA as well. The control said, “Cold Case Christianity is our gift to you in thanks for a donation to Harvest Ministries today. You can get yours by going to this page” The treatment said, “You can get a copy of Cold Case Christianity by going to this page”. They A/B split their email file and ran the test.
Research Question
Which call-to-action would get more people to click through to the donation page?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | CTRL - "donation" in CTA | 6.7% | ||
| T1: | TRT - No "donation" in CTA | 10.9% | 62.6% | 100.0% |
This experiment has a required sample size of 351 in order to be valid. Since the experiment had a total sample size of 59,966, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
62.6% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The treatment CTA had a 62.6% increase in clicks compared to the control. It is really encouraging to see such a significant increase in clicks; however, the ultimate goal is to get more donations. While we weren’t able to validate the donations, the initial data tells us that even though we had more people clicking through to the donation page, these people didn’t make a donation. The treatment had a decrease in donations by 9.3% (80% valid). The treatment CTA did reinforce the value proposition (which is why we had an increase in clicks), but it didn’t present clarity. When people clicked through from the email, they were probably expecting to just get the book/premium for free because there was no mention that they would get the premium when they gave a donation to Harvest Ministries. This shows the importance of being very clear in our CTA so that we don’t mislead people and they know what to expect during the next step in the giving process.