How a donation interruptor including a monthly upgrade ask affects recurring donation rates for previous donors

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Timeframe: 03/15/2023 - 03/25/2023

In an effort to increase recurring donor acquisition for a public policy organization during a recent high-urgency campaign, we wanted to measure the effect of interrupting the donation process with a pop-up asking the donor to become a monthly supporter instead of making only a one-time gift.

This pop-up was set up as an A/B test that would only fire for 50% of donors who were in the process of completing a one-time gift under $100, reiterating the value of becoming a monthly donor, both in terms of impact and exclusive benefits, and asking them to a) make a recurring monthly donation of $20 or b) proceed with their one-time gift.

This is a follow-up experiment to a previous donation interruptor test in order to test whether less body copy, and thus a better mobile viewing experience, would lead to an increase in recurring donations.

This A/B test targeted only the member donation page (previous donors of $25 or more).

Research Question

We believe that presenting a monthly upgrade ask for one-time, returning donors during the donation process will achieve higher recurring donation rates by increasing its visibility at the decision-making point and prominently communicating its value.

Design

C: Control

T1: Donation interrupter

Results

This experiment has a required sample size of 184 in order to be valid. Since the experiment had a total sample size of 1,330, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 20.9% decrease in conversion rate
× 0% increase in average gift

Key Learnings

The key learnings from this experiment are that interrupting the donation process with a pop-up asking donors to become monthly supporters instead of making a one-time donation result in a decrease in donations for all traffic.

After running this experiment for 9 days, we saw a 64.6% increase in monthly giving with a 81.6% level of confidence. As a result, this experiment is invalid.

We also saw a 20.9% decrease in overall donations with a 99.9% level of confidence.

We believe that multiple factors contributed to producing this result:

• Forcing donors to make another choice after submitting their one-time gift may produce friction. Asking donors to commit to a recurring gift in this case may be too strong of an ask and be perceived as pushy or invasive.
• Asking donors to join a Leadership Club may produce confusion at what membership entails, how it is different from “renewing” Heritage membership, and seem like too much of a commitment as opposed to a one-time gift to receive a Heritage 2023 member card.
• Results may be inaccurate due to the experiment recording all page viewers as part of the sample, even before they’ve seen either the control or donation interruptor experience. The sample size should be adjusted to apply only to those who have clicked the donation form submit button, excluding viewers who land on the page but navigate away before seeing either experience. This will isolate the experiment data only to those who donate or attempt to donate.

Based on these results and the data concerns, we will explore additional testing with added technical feature to record results only when the user clicks on the submit button and sees either experience at that step.