The Heritage Foundation

How highlighting that only a small minority people donate affects donor conversion rate

Experiment ID: #168351

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Timeframe: 01/19/2024 - 01/31/2024

The Heritage Foundation had been running a Voter Fraud survey for several months. The survey was incredibly cost-effecient at getting emails, but the donor conversion rate had been less than one percent.

They had seen other tests that highlighted the fact that “less than 1%” of people who take the survey are willing to take the next step and donate. This language had increased conversion because it appealed to the ideal donor, who was attracted to the idea of being in a small minority of people who were willing to do the “right thing” rather than the “popular thing”.

They inserted this language in two places, and split the traffic to determine a winner.

Research Question

We believe that highlighting that only a small minority people donate for survey takers will achieve an increase in conversion rate because it will appeal to the ideal donor, who are attracted to the idea of being in a small minority of people who were willing to do the “right thing” rather than the “popular thing”. because it will appeal to the ideal donor, who are attracted to the idea of being in a small minority of people who were willing to do the “right thing” rather than the “popular thing”.

Design

C: Control
T1: Less than 1% Copy

Results

 Treatment NameConv. RateRelative DifferenceConfidenceAverage Gift
C: Control 0.00%
T1: Less than 1% Copy 1.2%100.0% 100.0%

This experiment has a required sample size of 323 in order to be valid. Since the experiment had a total sample size of 5,214, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 100.0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The key learnings from this experiment are:

1. Highlighting that only a small minority of people donate for survey takers can significantly increase the conversion rate. The treatment group, which included this language, experienced a 105.4% increase in Transactions for All Traffic compared to the control group.

2. The use of language that appeals to the ideal donor, emphasizing the importance of doing the “right thing” rather than the “popular thing,” can be an effective strategy to attract donations. This was evidenced by the significant increase of 91.7% in Transactions – One-Time for the treatment group.

This is an application of social identity theory. This theory suggests that people tend to categorize themselves and others into various social groups. Supporting the underdog can be related to group identity and solidarity with a particular group.

Based on these learnings, it is recommended to continue using the language that highlights the small minority of people who donate in future experimentation. This messaging appears to resonate with the target audience and create a sense of exclusivity and righteousness in supporting the cause. However, it would also be beneficial to continuously test and optimize this messaging to ensure its continued effectiveness and relevance.


Experiment Documented by Jeff Giddens
Jeff Giddens is President of NextAfter.

Question about experiment #168351

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.