How personalization and brand reduction affect conversion on a post-survey donation page
Hillsdale College
Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.
Experiment Summary
Timeframe: 09/14/2020 - 10/17/2020
Hillsdale College had run an experiment during the pre-launch of their Introduction to Western Philosophy course that tested two variables independently. First, they tested using personalization in the headline of the “instant” donation page to increase the buy-in of the registrant and get them reading the page. Secondly, they noticed that the donation page had many mentions of Hillsdale College, which seemed redundant and seemed to get the brand “in the way” of the relationship that the copy was attempting to develop with the donor. They tested a separate treatment that did NOT include the personalization but did reduce the mentions of the brand, substituting pronouns where appropriate. These two treatments both appeared to lift revenue, 19% and 13% respectively. They wanted to re-test this, and combine the elements into one treatment. They chose their Electoral College Survey, developed the treatment, and launched a test.
Research Question
How will personalization and brand reduction affect conversion on a post-survey donation page?
Design
Results
| Treatment Name | Revenue per Visitor | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | $3.02 | $77.65 | ||
| T1: | T1 - Reduced Brand with Personalization | $3.43 | 13.4% | 99.8% | $79.58 |
This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was met so these experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 10.7% increase in conversion rate
× 2.5% increase in average gift
13.4% increase in revenue
Key Learnings
The treatment, which combined the personalization and the reduced brand mention, increased revenue by 13.4%. This reinforces the learning from the initial experiment, and shows that a similar lift in revenue holds for a treatment on a different offer, especially when combining the treatments.
This experiment results suggests that:
- Personalization has a statistically significant impact on conversion.
- Reducing the mentions of the brand on the donation page has a statistically significant impact on conversion.
- Both elements combined contribute to a lift in average gift and revenue per visitor.