How adding a call out to current Leaders Club donor affected revenue during Kickoff
Human Coalition
Experiment Summary
Timeframe: 01/20/2024 - 02/04/2024
Past testing has shown that adding a tool tip style message to a donation page with call out can have a directional lift in revenue. We decided to test this concept out with current Leaders Club donors who were being asked for a one-time gift during our kick-off campaign. The call out mentioned that one-time gifts from Leaders club members last year helped contribute to a record year in conservatives trained by The Heritage Foundation. The call out pointed to an upgraded amount from previous giving.
While this experiment did not validate we saw a strong directional lift in revenue on both desktop and mobile, but mobile saw a 108% increase in revenue from the control. Results for new and returning visitors were very close.
Asking for an upgrade in past giving while mentioning the impact that one-time gifts from Leaders Club members can have showed a strong case for future testing.
Research Question
We believe that adding a one-time gift call out focused on the increased impact of their one-time gift and focusing the ask on an upgrade amount for leaders club members will achieve a lift in average gift because donors see the value of their donation tied to specific tangible outcomes and are willing to contribute more to see similar outcomes.
Design
Results
| Treatment Name | Revenue per Visitor | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | $31.39 | $59.23 | ||
| T1: | Variation 1 | $47.37 | 50.9% | 80.5% | $108.27 |
This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was not met so these experiment results are not valid.
Key Learnings
The key learnings from this experiment are as follows:
1. When adding a one-time gift callout focused on increased impact and upgrading the ask amount, there was a significant lift in revenue. The treatment group saw a 50.9% increase in revenue compared to the control group.
2. The uplift in revenue was more prominent on mobile devices, with a 108% increase in revenue compared to the control group. This suggests that optimizing for mobile users can have a significant impact on donation revenue.
3. The experiment did not achieve a statistical level of confidence above 85%, which means the results are not considered valid. Therefore, the findings from this experiment should not be solely relied upon for future decision-making.
In future experimentation, the key learnings from this experiment can be used to inform new hypotheses and tests. The focus should be on creating a one-time gift callout that emphasizes the increased impact of the donation and offers an upgrade amount for leaders club members. Additionally, it is important to continue optimizing the donation process for mobile users to maximize revenue. It is recommended to run further experiments to validate the findings from this experiment and gather more data for conclusive decision-making.