How using a year-end specific sticky bar on acquisition pages impacted conversions
Hoover Institution
Experiment Summary
Timeframe: 12/20/2023 - 01/01/2024
The Hoover Institution is the public policy think-tank for Stanford University.
During the crucial year-end fundraising season, we wanted to see whether we could impact conversion rates on content offer instant donation pages by using a year-end specific sticky bar.
We ran this on all content offer instant donation pages during the last 10 days of the year, as that is when Hoover sees the most revenue come in online in December.
Research Question
We believe that adding a sticky bar with year-end messaging for content offer downloaders will achieve a change in conversion rates because the urgency behind year-end giving deadlines.. because the urgency behind year-end giving deadlines..
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | 6.4% | $65.12 | ||
| T1: | Sticky bar | 5.6% | -12.2% | 79.8% | $134.33 |
This experiment has a required sample size of 7,034 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
The key learning from this experiment is that adding a sticky bar with year-end messaging for content offer downloaders did not achieve the desired change in conversion rates. The treatment group actually saw a decrease in transactions by 12.2% compared to the control group.
It is important to note that the statistical level of confidence for these results is only 79.8%, which is below the generally accepted minimum threshold of 85%. This means that the results are not considered valid and should not be used for future decision making.
While transaction conversion rates were lower for the treatment group, when looking at the impact on revenue, we see that the treatment version had a 104% higher average gift. Given the time of year this experiment was run, this is most likely due to a larger gift skewing results, but it is still interesting that the sticky bar with year-end messaging motivated that donor to make that large gift.