The Heritage Foundation

How a tabbed donation form impacts recurring donor acquisition

Experiment ID: #26540

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Timeframe: 12/13/2019 - 01/09/2020

During year end, we wanted to find ways to increase membership to the Leaders Club for The Heritage Foundation. One tactic that had worked with a similar organization was utilizing a tabbed donation form. This allowed us to adjust the suggested gift array lower for recurring donors and changed the order of the mental conversation to ask whether they wanted to give recurring prior to asking how much they wanted to give.

Research Question

We believe that using a tabbed form for visitors to /renew and the main donation page will achieve increased recurring donor acquisition.

Design

C: Control
T1: Tabbed Recurring Gifts

Results

  Treatment Name Conv. Rate Relative Difference Confidence Average Gift
C: Control 0.80% $0.00
T1: Tabbed Recurring Gifts 9.5% 1,090.5% 100.0% $0.00

This experiment has a required sample size of 50 in order to be valid. Since the experiment had a total sample size of 3,788, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 1,090.5% increase in conversion rate
× 0% increase in average gift

Key Learnings

We observed a 1,029% increase to recurring donor acquisition with the treatment. However, after a deeper analysis, we discovered that many of these donor didn’t realize they had been making a recurring gift. This is due to the recurring option being pre-selected for all visitors.

Since we had no way of knowing what gift type they were trying to give, we decided to cancel all recurring gifts and rerun the experiment in the new year without selecting the recurring option for them.

Notable changes to other metrics due to the experiment:

  • Donations experienced a 22.2% decrease with a 100% level of confidence.
  • Revenue experienced a 35.9% decrease with a 98% level of confidence.

Observations of the impact on other visitor segments in the experiment:

  • Desktop visitors had a 1,662.5% increase in monthly recurring gift with a 100% level of confidence.
  • Mobile visitors had a 1,609.8% increase in monthly recurring gift with a 100% level of confidence.
  • New visitors had an 8,762.1% increase in monthly recurring gift with a 100% level of confidence.
  • Returning visitors had a 572.5% increase in monthly recurring gift with a 100% level of confidence.
  • Organic Search visitors had a 10.6% increase in monthly recurring gift with a 99% level of confidence.


Experiment Documented by NextAfter

Question about experiment #26540

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.