How removing a premium offer affects conversion Experiment ID: #6798

Hillsdale College

Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.

Experiment Summary

Timeframe: 4/18/2017 - 6/12/2017

Hillsdale College offered premiums as incentives to donate for many of their free online courses. These premiums helped keep average gift high and made the courses generate donation revenue for the College.

But the team wondered if the premium ever got in the way of the donation. If the prospective donor was not interested in the premium offer, did they abstain from giving entirely? Were prospective donors confusing the offer with the incentive? They decided to test this on their course “The Presidency and the Constitution” by drafting a page that removed the premium book offer, in this case, their popular “Constitution Reader”.

They split traffic to the donation page and launched an A/B test to see how it affected revenue and donor conversion.

Research Question

How does removing the premium offer affect conversion and revenue?

MECLABS Conversion Factors Targeted

C = 4m + 3v + 2( i - f) - 2a ©

Copyright 2015, MECLABS

Design

C: Control
T1: No Premium

Results

Treatment Name Conv. Rate Relative Difference Confidence Average Gift
C: Control 6.2% $54.72
T1: No Premium 6.8% 11.1% 85.1% $42.23

This experiment has a required sample size of 9,973 in order to be valid. Since the experiment had a total sample size of 10,838, and the level of confidence is not above 95% the experiment results are not valid.

Key Learnings

The treatment page without the premium offer produced an 11% lift in conversion, though this test failed to gain statistical validity. This provided a directional learning that removing the premium offer might increase conversion some of the time. However, the page without the premium also reduced average gift by $12, which lowered overall revenue.

If this test had validated, Hillsdale would have had to choose between more donors or more dollars. This test prompted another test on a different course to ensure that the finding wasn’t specific to this course.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Jeff Giddens

Jeff is a Senior Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.