The impact of moving the donation ask earlier in the funnel
Alliance Defending Freedom
Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.
Experiment Summary
Timeframe: 06/14/2018 - 06/29/2018
The blog for Alliance Defending Freedom get significant traffic on a daily basis. Over the past several months, we had been able to test into a series of new offers that had lifted the email acquisition rate from less than 1% to 4% – 6%. This meant several hundred new emails were coming on to the file on a weekly basis and each of these subscribers were shown a donation page once they submitted their email. This meant a certain percentage of them would convert to donors instantly.
ADF had recently developed a new embeddable donation form that we wanted to test out on the blog. Historical tests had found that, when linking to a donation form, the blog did not serve as a significant source for donations. We wanted to determine if moving the ability to donate directly on to the blog would allow us to reduce the friction associated with the donation funnel and, thereby, increase donor acquisition.
Research Question
Will moving the donation ask earlier in the funnel increase overall donor conversion?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | NIFLA Statement of Belief | 0.02% | ||
| T1: | Inline Donation Form | 0.07% | 263.9% | 91.2% |
This experiment has a required sample size of 12,952 in order to be valid. Since the experiment had a total sample size of 20,088, and the level of confidence is not above 95% the experiment results are not valid.
Key Learnings
Upon first glance, the embedded donation form appeared to have a 60% decrease to overall conversion. However, when digging deeper into the numbers, we discovered that while there was a 60% decrease between the people that saw the treatment compared to the control, the actual conversions that came from the email acquisition offer compared to the donation offer told a different story. If we looked at just those conversions, the direct donation ask had a 264% increase (at 91% level of confidence).
This variance is the result of the high converting traffic that occurs during the fiscal year end period. Essentially, subscribers would read a blog post and they later got an email asking them to make a donation (which was unrelated to their actions on the blog post). However, since they saw the original blog post, their donation was then attributed to the experiment.
What we can take away from this experiment is that we always need to dig deeper into the top level results reported by the various systems we use. We also now know that we need to re-test this concept once the fiscal year has passed.