How the addition of Apple Pay impacted donor conversion - NextAfter
The Heritage Foundation

How the addition of Apple Pay impacted donor conversion

Experiment ID: #6509

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Timeframe: 02/24/2017 - 04/18/2017

The Heritage Foundation is consistently looking for new ways to make the donation process both easier and better for their members. The Apple Pay option had recently gone mainstream with its incorporation into both iPhones and Mac web browsers. This one-click giving option made it significantly easier for visitors that had the service set up; for those that did not, they would not have seen any difference between the forms.

However, before rolling out the new feature, we wanted to measure its impact on the donor audience. Since this new feature was only applicable to Safari users on a Mac, we targeted the experiment to that audience.

Research Question

Will the addition of Apple Pay decrease the friction associated with the donation process enough to increase donor conversion?

Design

C: No Apple Pay
T1: Apple Pay

Results

  Treatment Name Conv. Rate Relative Difference Confidence
C: No Apple Pay 5.6%
T1: Apple Pay 5.5% -1.6% 32.3%

This experiment has a required sample size of 523,631 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

The new Apple Pay feature did not have any statistically significant impact on donor conversion or revenue. Our hypothesis is that this is likely due to the slow adoption of Apple Pay for Mac and Apple users. The portion of users that will be implementing Apple Pay will only be increasing in coming month so we decided to roll out the new form for visitors moving forward.


Experiment Documented by NextAfter

Question about experiment #6509

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.