How reducing the options on a gift array impacted donations
Canadian Taxpayers Federation
A not-for-profit citizen's advocacy group dedicated to lower taxes, less waste and accountable government.
Experiment Summary
Timeframe: 02/16/2023 - 03/09/2023
Because we had seen promising results on a similar test, we wanted to test reducing the number of options on a gift array for an organization’s French-speaking audience to see if they would also have an increase in conversion. The control donation page offered 5 donation amounts and an optional “Other” box to choose a gift of any amount. The treatment presented only a $10 ask and the “Other” field. We ran the test for 3 weeks splitting traffic evenly and measured results on donations and revenue.
Research Question
We believe that reducing the number of gift array options for instant donation page visitors will achieve an increase in donor conversion because there is less friction in the decision-making process.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | 4.5% | $0.00 | ||
| T1: | $10 | 9.7% | 116.3% | 96.7% | $0.00 |
This experiment has a required sample size of 189 in order to be valid. Since the experiment had a total sample size of 426, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 116.3% increase in conversion rate
× 0% increase in average gift
Key Learnings
What we found after running the test for 3 weeks was the treatment produced a 116% lift in donations compared to the control with 96% level of confidence so the results are valid. We also saw a 136% lift in revenue with 98% level of confidence. However, we still had less than the minimum number of required transactions for these results to be statistically significant. We believe these results are due to the decrease in decision making because the donor does not have to spend additional time deciding how much to give. In future testing, we will want to test other gift amounts presented as an isolated option to see if the low $10 amount was the reason for the increase or if it was solely a result of reducing the number of options.