How a defense-minded approach with an expanded value proposition affects conversion rate
Hillsdale College
Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.
Experiment Summary
Timeframe: 06/09/2020 - 08/08/2020
Hillsdale College wanted to test the copy on the donation page for their online courses site. The control copy was a positive message that put power in the donor’s hands—”You can help educate America”. They hypothesized that a stronger posture that explained the “why” behind the importance of education—defending liberty. The copy also did more to explain the problem to which Hillsdale’s educational efforts provides a solution, and gave extra descriptive copy of the premium. They launched an A/B test to determine a winner.
Research Question
How will a defense-minded approach with an expanded value proposition affect conversion rate?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | 18.6% | $0.00 | ||
| T1: | Treatment | 16.5% | -11.2% | 95.6% | $0.00 |
This experiment has a required sample size of 2,544 in order to be valid. Since the experiment had a total sample size of 5,349, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 11.2% decrease in conversion rate
× 0% increase in average gift
Key Learnings
The treatment, with the extended value proposition and defense-minded headline, decreased conversion rate by 11.5%. This showed that a positive message in this case (plus the benefit of shorter copy) was more likely to inspire a gift.
This could reflect the mindset of donors while they are on the course platform—they are not thinking about defense, but opportunity. Further testing might expand opportunity there.