Dallas Theological Seminary

How removing creative elements affects conversion

Experiment ID: #2806

Dallas Theological Seminary

The DTS mission is, “to glorify God by equipping godly servant-leaders for the proclamation of His Word and the building up of the body of Christ worldwide.” They strive to help men and women fulfill the Great Commission and the Great Commandment, or more simply: Teach Truth. Love Well.

Experiment Summary

Timeframe: 11/18/2015 - 12/07/2015

Dallas Theological Seminary launched a free online course studying the Gospel of John that was enrolling students at a rapid clip. DTS wanted to see if removing creative elements would improve the clarity of the offer by allowing the eye to focus on the headline, which delivered the value proposition. They also wanted to test a different headline that used another word than “Study”, which conveyed work to the visitor.

So they launched an experiment to determine whether streamlining the page would improve clarity and increase visitor-to-student conversion.

Research Question

Will removal of creative elements like the graphical banner improve clarity and increase conversion?

Design

C: With Banner / Gospel of John
T1: No Banner / Free Course

Results

 Treatment NameConv. RateRelative DifferenceConfidenceAverage Gift
C: With Banner / Gospel of John 1.1%$88.72
T1: No Banner / Free Course 0.89%-18.1% 95.4%$72.89

This experiment has a required sample size of 19,349 in order to be valid. Since the experiment had a total sample size of 40,080, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 18.1% decrease in conversion rate
× 17.8% decrease in average gift
32.8% decrease in revenue

Key Learnings

Removing the banner not only decreased conversion by nearly 20%, it also directionally reduced the average gift of those who signed up and then made a donation.

This reminded us that, while clarity is an important factor in the value proposition, appeal and credibility are essential as well. In this case, the graphical elements seem to add credibility to the course.

In addition, while the use of the word “study” might convey work, it also seems to contribute to the credibility of the course, and may align with a key motivation of visitors to the page.

Though other experiment have shown that removing creative elements from the page might improve conversion, this instance proves yet again that what might be a “best practice” for one landing page must be tested on another.


Experiment Documented by Jeff Giddens
Jeff Giddens is President of NextAfter.

Question about experiment #2806

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.