Dallas Theological Seminary

How adding a gift array affects conversion

Experiment ID: #2802

Dallas Theological Seminary

The DTS mission is, “to glorify God by equipping godly servant-leaders for the proclamation of His Word and the building up of the body of Christ worldwide.” They strive to help men and women fulfill the Great Commission and the Great Commandment, or more simply: Teach Truth. Love Well.

Experiment Summary

Timeframe: 11/30/2015 - 12/07/2015

Dallas Theological Seminary launched a free online course to grow their email file and attract new donors. After a visitor signed up for the course, they were presented with an opportunity to give, accompanied by a premium offer to receive a Bible commentary.

Previous testing and optimization with DTS had shown that their donors preferred an open field instead of a gift array with recommended gift amounts. But as this course began to take off, they realized they were bringing new people to the page. They hypothesized that these new people might benefit from a choice of gift options rather than a blank field.

So they created a second page with the same copy, but three gift options above the donation form and ran an A/B split test to find out which one resulted in more conversions and donation revenue.

Research Question

Do suggested gift amounts convert better than an open field?

Design

C: Open Field
T1: Gift Array

Results

 Treatment NameConv. RateRelative DifferenceConfidenceAverage Gift
C: Open Field 2.1%$90.71
T1: Gift Array 2.8%34.7% 95.7%$75.81

This experiment has a required sample size of 3,542 in order to be valid. Since the experiment had a total sample size of 7,556, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 34.7% increase in conversion rate
× 16.4% decrease in average gift
12.6% increase in revenue

Key Learnings

The treatment with the gift array produced a 34.9% lift in conversion. However, it also produced a 21% decrease in average gift. Since the decrease in average gift was less than the increase in conversion, the treatment also generated more revenue, which made it the winner.

This tells us that the “unlimited choice” of an open field produces too much decision friction in this instance. It also reminds us that as an audience grows past those who are familiar with an organization, previously tested ideas might be called into question — which reinforces the value and importance of continual testing and optimization.


Experiment Documented by Jeff Giddens
Jeff Giddens is President of NextAfter.

Question about experiment #2802

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.