How decision friction impacts donor conversion Experiment ID: #17181

Billy Graham Evangelistic Association

Experiment Summary

Timeframe: 6/28/2019 - 7/28/2019

On BGEA’s primary donation page, they offer people 14 different gift designations to choose from during the first step in their donation process. In an effort to optimize the giving experience and increase donor conversion, we previously tested removing all of the gift designation hypothesized narrowing down the number of gift designation options to the top five. Using historical giving data, we pulled the top five designations and developed a treatment that featured just those five. People would still be able to see the other 9 options if they wanted by clicking the “See more>>” link. We also developed another variation of the treatment that emphasized the top giving option as the “Most Urgent” need and removed the imagery and copy from the rest of the gift designations.

Research Question

Would we be able to increase donor conversion by removing some of the gift designations?

Design

C: Control
T1: Top 5 Designation List
T2: Top 5 List with Urgent Need

Results

Treatment Name Conv. Rate Relative Difference Confidence Average Gift
C: Control 42.5% $0.00
T1: Top 5 Designation List 47.3% 11.4% 99.9% $0.00
T2: Top 5 List with Urgent Need 42.0% -1.1% 24.7% $0.00

This experiment has a required sample size of 1,008 in order to be valid. Since the experiment had a total sample size of 7,028, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 11.4% increase in conversion rate
× 0% increase in average gift

Key Learnings

After running this experiment for a month, we were able to increase overall donor conversion by 11.4% by narrowing down the gift designation options down from 14 to 5. Since this experiment was run on the primary donation page where motivations and audience can vary greatly, we analyzed all devices (mobile and desktop) and user types (new and returning). Regardless of device or user, donor conversion increased with the first treatment. The second treatment had differing results based on device and audience, but none of those results had high validity.

This experiment tells us that there was decision friction on the page in the number of designations and options we were presenting people. By reducing the numbers of options, we were able to increase donor conversion and overall revenue for the organization.

This page gets a significant volume of traffic to it. The potential impact that this experiment can have on revenue moving forward is substantial. In fact, based on historical giving, when the winning treatment is rolled out to 100% of the traffic over the next three months, it has the potential to bring in an additional projected $50k in revenue!


Experiment Documented by...

Courtney Gaines

Courtney is the Senior Director of Optimization at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.