How a “most popular” pre-selected value in the gift array did not hurt conversion or revenue Experiment ID: #18716

The Fund for American Studies

Experiment Summary

Timeframe: 10/4/2019 - 11/4/2019

In an attempt to increase the average gift for The Fund for American Studies’ acquisition campaign, we decided to experiment with a “most popular” treatment on the $50 amount in the gift array to see if it would increase the average gift amount from instant donors.

Research Question

Would a visual treatment using that increases the visibility of the pre-selected value increase the average gift from instant donors?

Design

C: Control
T1: Most Popular Treatment

Results

Treatment Name Conv. Rate Relative Difference Confidence Average Gift
C: Control 4.5% $0.00
T1: Most Popular Treatment 5.0% 10.9% 39.2% $0.00

This experiment has a required sample size of 14,406 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

Although we did not see a validated increase in either donor conversion rate or revenue, we observed a directional lift in both metrics.

The key learning with this experiment was that after running for a month, this treatment DID NOT show, at any point, a decrease in either one of these metrics, which was enough for us to consider that the lifts we achieved were directional, and that there would not be risk in rolling this treatment out as the “new control” version of the donation page.


Experiment Documented by...

Greg Colunga

Greg is Executive Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.