How personalization and brand reduction affect donor conversion on a post-signup donation page
Hillsdale College
Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.
Experiment Summary
Timeframe: 08/07/2020 - 09/05/2020
Hillsdale College was running a pre-registration campaign to promote their new course “Introduction to Western Philosophy”. They wanted to test two concepts on the donation page that followed the pre-registration. The first concept they wanted to test was personalization. Since the user signed up and gave their first name, they wanted to test a version of the donation page that congratulated the user by name for signing up. They were interested to see if this personalization would increase engagement on the page and humanize the “transaction”, resulting in an increased conversion rate. They also noticed that the approved copy had multiple repeated mentions of both Hillsdale and the full course name, “Introduction to Western Philosophy”. They hypothesized that removing these repeat mentions, replacing with “our” and “this course” would both “get the brand out of the way” and increase the readability of the page, resulting in increased conversion. Since they expected a large influx of traffic for this pre-registration period, they launched a three-way test to see if they could find a winner.
Research Question
How will personalization and brand reduction affect donor conversion on a post-signup donation page?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | 8.8% | $0.00 | ||
| T1: | T1 - Personalization | 9.0% | 2.6% | 29.2% | $0.00 |
| T2: | T2 - New copy | 9.7% | 10.6% | 86.9% |
This experiment has a required sample size of 10,973 in order to be valid. Since the experiment had a total sample size of 13,125, and the level of confidence is not above 95% the experiment results are not valid.
Key Learnings
Both treatments had positive increases on the control, but in different ways.
Treatment one, with the added personalization, actually had very little impact on conversion but resulted in a statistically valid 26% lift in revenue. This indicates that personalization may increase the force of the value proposition in the mind of the donor, inspiring them to give more generously.
Treatment two, which removed the brand and course mentions, produced two increases, neither statistically valid but both directionally interesting. First, it increased conversion rate by 10.6% with a 86.9% LoC. While this did not validate, it did show a trend that, with more traffic, might be proven favorable. It also produced a 19% lift in revenue, with a 91% LoC—again, not statistically significant, but revealing a learning to be re-tested in a future campaign.
Hillsdale’s team decided to combine the two treatments in a future offer to see if the combination of the personalization (which had a valid lift to revenue) and the reduction of brand terminology (which had a near-valid lift to conversion) would work together to produce a valid lift overall.