How the addition of Apple Pay impacted donor conversion
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Timeframe: 02/24/2017 - 04/18/2017
The Heritage Foundation is consistently looking for new ways to make the donation process both easier and better for their members. The Apple Pay option had recently gone mainstream with its incorporation into both iPhones and Mac web browsers. This one-click giving option made it significantly easier for visitors that had the service set up; for those that did not, they would not have seen any difference between the forms.
However, before rolling out the new feature, we wanted to measure its impact on the donor audience. Since this new feature was only applicable to Safari users on a Mac, we targeted the experiment to that audience.
Research Question
Will the addition of Apple Pay decrease the friction associated with the donation process enough to increase donor conversion?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | No Apple Pay | 5.6% | ||
| T1: | Apple Pay | 5.5% | -1.6% | 32.3% |
This experiment has a required sample size of 523,631 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
The new Apple Pay feature did not have any statistically significant impact on donor conversion or revenue. Our hypothesis is that this is likely due to the slow adoption of Apple Pay for Mac and Apple users. The portion of users that will be implementing Apple Pay will only be increasing in coming month so we decided to roll out the new form for visitors moving forward.