How removing creative elements affects conversion
Dallas Theological Seminary
The DTS mission is, “to glorify God by equipping godly servant-leaders for the proclamation of His Word and the building up of the body of Christ worldwide.” They strive to help men and women fulfill the Great Commission and the Great Commandment, or more simply: Teach Truth. Love Well.
Experiment Summary
Timeframe: 11/18/2015 - 12/07/2015
Dallas Theological Seminary launched a free online course studying the Gospel of John that was enrolling students at a rapid clip. DTS wanted to see if removing creative elements would improve the clarity of the offer by allowing the eye to focus on the headline, which delivered the value proposition. They also wanted to test a different headline that used another word than “Study”, which conveyed work to the visitor.
So they launched an experiment to determine whether streamlining the page would improve clarity and increase visitor-to-student conversion.
Research Question
Will removal of creative elements like the graphical banner improve clarity and increase conversion?
Design
Results
Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
---|---|---|---|---|---|
C: | With Banner / Gospel of John | 1.1% | $88.72 | ||
T1: | No Banner / Free Course | 0.89% | -18.1% | 95.4% | $72.89 |
This experiment has a required sample size of 19,349 in order to be valid. Since the experiment had a total sample size of 40,080, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 18.1% decrease in conversion rate
× 17.8% decrease in average gift
32.8% decrease in revenue
Key Learnings
Removing the banner not only decreased conversion by nearly 20%, it also directionally reduced the average gift of those who signed up and then made a donation.
This reminded us that, while clarity is an important factor in the value proposition, appeal and credibility are essential as well. In this case, the graphical elements seem to add credibility to the course.
In addition, while the use of the word “study” might convey work, it also seems to contribute to the credibility of the course, and may align with a key motivation of visitors to the page.
Though other experiment have shown that removing creative elements from the page might improve conversion, this instance proves yet again that what might be a “best practice” for one landing page must be tested on another.
Question about experiment #2806
If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.