How removing a premium offer affects conversion Experiment ID: #6798
Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.
Timeframe: 4/18/2017 - 6/12/2017
Hillsdale College offered premiums as incentives to donate for many of their free online courses. These premiums helped keep average gift high and made the courses generate donation revenue for the College.
But the team wondered if the premium ever got in the way of the donation. If the prospective donor was not interested in the premium offer, did they abstain from giving entirely? Were prospective donors confusing the offer with the incentive? They decided to test this on their course “The Presidency and the Constitution” by drafting a page that removed the premium book offer, in this case, their popular “Constitution Reader”.
They split traffic to the donation page and launched an A/B test to see how it affected revenue and donor conversion.
How does removing the premium offer affect conversion and revenue?
MECLABS Conversion Factors Targeted
C = 4m + 3v + 2( i - f) - 2a ©
Copyright 2015, MECLABS
|Treatment Name||Conv. Rate||Relative Difference||Confidence||Average Gift|
This experiment has a required sample size of 9,973 in order to be valid. Since the experiment had a total sample size of 10,838, and the level of confidence is not above 95% the experiment results are not valid.
The treatment page without the premium offer produced an 11% lift in conversion, though this test failed to gain statistical validity. This provided a directional learning that removing the premium offer might increase conversion some of the time. However, the page without the premium also reduced average gift by $12, which lowered overall revenue.
If this test had validated, Hillsdale would have had to choose between more donors or more dollars. This test prompted another test on a different course to ensure that the finding wasn’t specific to this course.