How adding the persons first name vs just saying Thank You on the instant donation page can affect the conversion rate
Hoover Institution
Experiment Summary
Timeframe: 06/14/2021 - 07/18/2021
In this experiment, we wanted to test which would convert better in the header of the instant donation page: Thank you, friend (the control), Thank You, First Name (treatment 2), or Thank you! (Treatment 1)
We tested this with an eBook offer that ran on Facebook as well as sent to their housefile. We have run a similar experiment in the past where we tested Thank you, friend vs Thank you, First Name and saw that adding the personalization aspect to it led to a higher conversion rate. We wanted to run this as a 3-way test. The hypotheses being that just saying thank you will help increase conversion rate as donors value privacy and when they see their first name it could be turn off or in some way feel like a valuation of that privacy since they just provide their first name.
Research Question
We believe that by limiting the header to just saying Thank You for those that visit the instant donation page will achieve a better conversion rate.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | 3.2% | $0.00 | ||
| T1: | Thank you! | 5.4% | 69.7% | 97.2% | $0.00 |
| T2: | Thank you, [First Name] | 4.1% | 28.4% | 67.4% | $0.00 |
This experiment has a required sample size of 952 in order to be valid. Since the experiment had a total sample size of 2,463, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 69.7% increase in conversion rate
× 0% increase in average gift
Key Learnings
My hypothesis is that this has to do with Hoover donors and privacy. When we say thank you friend it is very un-personal and has shown to be a turn-off. However, when we add that personal aspect vs Thank You, we saw that keeping in simple works better. I think when we add their first name it could make the potential donor feel like a valuation of that privacy since they just provide their first name and they are now seeing it on the donation page.
We are testing this same test again on a different offer to see if we get similar results across the board.
Notable changes to other metrics due to the experiment:
- Revenue experienced a 79.2% increase with a 97% level of confidence.
Observations of the impact on other visitor segments in the experiment:
- Mobile visitors had a 91.8% increase in donations with a 93% level of confidence.
- Returning visitors had a 180.0% increase in donations with a 97% level of confidence.
- (Other) visitors had a 66.4% increase in donations with a 94% level of confidence.
- Male visitors had a 183.3% increase in donations with a 93% level of confidence.
- 65+ visitors had a 335.6% increase in donations with a 91% level of confidence.