How adding a donors first name to the instant donation page can increase conversion rate
Hoover Institution
Experiment Summary
Timeframe: 01/28/2021 - 05/01/2021
In this experiment, we added a donor’s first name to the instant donation page once they downloaded the eBook it said “Thank you, First name” while the control just said, “Thank you, friend”. This experiment was run specifically on Facebook and we had a 50/50 split on who saw the control vs the treatment.
The main metric we were looking to track on this experiment was the donor conversion rate. Previously with Hoover we ran this experiment and saw a lift but we didn’t launch it across the board so we wanted to test it again as they had also recently changed their donation platform.
Research Question
We believe that by adding the first name for those that visit the page will achieve an increase in both donations and revenue.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | 8.4% | $0.00 | ||
| T1: | First Name | 9.4% | 12.2% | 67.7% | $0.00 |
This experiment has a required sample size of 5,932 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
We had to stop this experiment early so it did not receive the necessary traffic it needed to validate (67%LoC) but with it seeing a 12% increase in conversion rate and a 43% increase in revenue (99% LoC) we will continue to test this across multiple Hoover offers.
People enjoy seeing their name and it adds that personal aspect and helps reinforce that they completed the signup.
Notable changes to other metrics due to the experiment:
- Revenue experienced a 43.8% increase with a 99% level of confidence.
Observations of the impact on other visitor segments in the experiment:
- Organic Search visitors had an 8.3% increase in donations with an 86% level of confidence.
- Paid Search visitors had a 14.3% increase in donations with an 87% level of confidence.
- 55-64 visitors had a 322.3% increase in donations with a 99% level of confidence.