How adding a gift array affects conversion
Dallas Theological Seminary
The DTS mission is, “to glorify God by equipping godly servant-leaders for the proclamation of His Word and the building up of the body of Christ worldwide.” They strive to help men and women fulfill the Great Commission and the Great Commandment, or more simply: Teach Truth. Love Well.
Experiment Summary
Timeframe: 11/30/2015 - 12/07/2015
Dallas Theological Seminary launched a free online course to grow their email file and attract new donors. After a visitor signed up for the course, they were presented with an opportunity to give, accompanied by a premium offer to receive a Bible commentary.
Previous testing and optimization with DTS had shown that their donors preferred an open field instead of a gift array with recommended gift amounts. But as this course began to take off, they realized they were bringing new people to the page. They hypothesized that these new people might benefit from a choice of gift options rather than a blank field.
So they created a second page with the same copy, but three gift options above the donation form and ran an A/B split test to find out which one resulted in more conversions and donation revenue.
Research Question
Do suggested gift amounts convert better than an open field?
Design
Results
Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
---|---|---|---|---|---|
C: | Open Field | 2.1% | $90.71 | ||
T1: | Gift Array | 2.8% | 34.7% | 95.7% | $75.81 |
This experiment has a required sample size of 3,542 in order to be valid. Since the experiment had a total sample size of 7,556, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 34.7% increase in conversion rate
× 16.4% decrease in average gift
12.6% increase in revenue
Key Learnings
The treatment with the gift array produced a 34.9% lift in conversion. However, it also produced a 21% decrease in average gift. Since the decrease in average gift was less than the increase in conversion, the treatment also generated more revenue, which made it the winner.
This tells us that the “unlimited choice” of an open field produces too much decision friction in this instance. It also reminds us that as an audience grows past those who are familiar with an organization, previously tested ideas might be called into question — which reinforces the value and importance of continual testing and optimization.
Question about experiment #2802
If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.