The Gospel Coalition

How adding a tabbed form on an acquisition offer impacted overall giving

Experiment ID: #89457

The Gospel Coalition

Experiment Summary

Timeframe: 03/25/2022 - 04/29/2022

Because we have found recurring donors to consistently have a higher lifetime value as well as a higher retention rate than one time donors, we wanted to identify additional ways to convert donation page visitors into recurring donors. The standard donation form control has a checkbox underneath the gift array that says “Make this a monthly gift” but our hypothesis was that asking for the gift amount first was causing friction to then commit the chosen amount as a monthly gift after the fact. We wanted to test introducing a tabbed form on an acquisition offer so we could determine if a tabbed form would have any substantial negative impact on overall giving before we tried to put this option on the main TGC donation page. We split traffic 50/50 between the two pages, making only the change to the donation form presentation of one time vs. monthly giving.

Research Question

We believe that offering a tabbed form for instant donation page visitors will achieve an increase in monthly donations.

Design

C: Control
T1: Treatment

Results

 Treatment NameConv. RateRelative DifferenceConfidenceAverage Gift
C: Control 0.44%$0.00
T1: Treatment 1.8%305.8% 99.5%$0.00

This experiment has a required sample size of 473 in order to be valid. Since the experiment had a total sample size of 1,861, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 305.8% increase in conversion rate
× 0% increase in average gift

Key Learnings

Unfortunately, after running the test for one month, zero recurring donations were made on either the control or the treatment. However, we were still able to analyze the one-time donations on the two versions. What happened was a 305.8% increase in donor conversion rate on the treatment with 99.4% level of confidence. As a result, revenue also experienced a 253.3% increase with a 97% level of confidence on the treatment page. Lifts were consistent across both desktop and mobile visitors alike so we can move forward with testing the tabbed form experience in an attempt to increase recurring giving on the main donation form without concern that there will be a negative impact or decrease in overall conversion. On future tests, we will look for ways to highlight the value proposition for monthly giving in the body copy to continue to aim for increased recurring donors. It is important to note that we did not reach the desired minimum of 50 transactions to fully validate this test, so we will need to re-run this on another offer where we can get additional traffic and volume of conversions to know for sure if the treatment is producing a true lift.


Experiment Documented by Rebekah Josefy
Rebekah Josefy is an Optimization Director at NextAfter.

Question about experiment #89457

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.