The impact of moving the donation ask earlier in the funnel Experiment ID: #9142

Alliance Defending Freedom

Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.

Experiment Summary

Timeframe: 6/14/2018 - 6/29/2018

The blog for Alliance Defending Freedom get significant traffic on a daily basis. Over the past several months, we had been able to test into a series of new offers that had lifted the email acquisition rate from less than 1% to 4% – 6%. This meant several hundred new emails were coming on to the file on a weekly basis and each of these subscribers were shown a donation page once they submitted their email. This meant a certain percentage of them would convert to donors instantly.

ADF had recently developed a new embeddable donation form that we wanted to test out on the blog. Historical tests had found that, when linking to a donation form, the blog did not serve as a significant source for donations. We wanted to determine if moving the ability to donate directly on to the blog would allow us to reduce the friction associated with the donation funnel and, thereby, increase donor acquisition.

Research Question

Will moving the donation ask earlier in the funnel increase overall donor conversion?

Design

C: NIFLA Statement of Belief
T1: Inline Donation Form

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: NIFLA Statement of Belief 0.02%
T1: Inline Donation Form 0.07% 263.9% 91.2%

This experiment has a required sample size of 12,952 in order to be valid. Since the experiment had a total sample size of 20,088, and the level of confidence is not above 95% the experiment results are not valid.

Key Learnings

Upon first glance, the embedded donation form appeared to have a 60% decrease to overall conversion. However, when digging deeper into the numbers, we discovered that while there was a 60% decrease between the people that saw the treatment compared to the control, the actual conversions that came from the email acquisition offer compared to the donation offer told a different story. If we looked at just those conversions, the direct donation ask had a 264% increase (at 91% level of confidence).

This variance is the result of the high converting traffic that occurs during the fiscal year end period. Essentially, subscribers would read a blog post and they later got an email asking them to make a donation (which was unrelated to their actions on the blog post). However, since they saw the original blog post, their donation was then attributed to the experiment.

What we can take away from this experiment is that we always need to dig deeper into the top level results reported by the various systems we use. We also now know that we need to re-test this concept once the fiscal year has passed.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.