How highlighting that only a small minority people donate affects donor conversion rate
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Timeframe: 01/19/2024 - 01/31/2024
The Heritage Foundation had been running a Voter Fraud survey for several months. The survey was incredibly cost-effecient at getting emails, but the donor conversion rate had been less than one percent.
They had seen other tests that highlighted the fact that “less than 1%” of people who take the survey are willing to take the next step and donate. This language had increased conversion because it appealed to the ideal donor, who was attracted to the idea of being in a small minority of people who were willing to do the “right thing” rather than the “popular thing”.
They inserted this language in two places, and split the traffic to determine a winner.
Research Question
We believe that highlighting that only a small minority people donate for survey takers will achieve an increase in conversion rate because it will appeal to the ideal donor, who are attracted to the idea of being in a small minority of people who were willing to do the “right thing” rather than the “popular thing”. because it will appeal to the ideal donor, who are attracted to the idea of being in a small minority of people who were willing to do the “right thing” rather than the “popular thing”.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | 0.57% | $35.00 | ||
| T1: | Less than 1% Copy | 1.2% | 105.4% | 98.0% | $34.35 |
This experiment has a required sample size of 1,834 in order to be valid. Since the experiment had a total sample size of 5,214, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 105.4% increase in conversion rate
× 1.8% decrease in average gift
101.7% increase in revenue
Key Learnings
The key learnings from this experiment are:
1. Highlighting that only a small minority of people donate for survey takers can significantly increase the conversion rate. The treatment group, which included this language, experienced a 105.4% increase in Transactions for All Traffic compared to the control group.
2. The use of language that appeals to the ideal donor, emphasizing the importance of doing the “right thing” rather than the “popular thing,” can be an effective strategy to attract donations. This was evidenced by the significant increase of 91.7% in Transactions – One-Time for the treatment group.
This is an application of social identity theory. This theory suggests that people tend to categorize themselves and others into various social groups. Supporting the underdog can be related to group identity and solidarity with a particular group.
Based on these learnings, it is recommended to continue using the language that highlights the small minority of people who donate in future experimentation. This messaging appears to resonate with the target audience and create a sense of exclusivity and righteousness in supporting the cause. However, it would also be beneficial to continuously test and optimize this messaging to ensure its continued effectiveness and relevance.