The Heritage Foundation

How adding membership level indicators affects membership renewal

Experiment ID: #447

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Timeframe: 01/07/2015 - 01/08/2015

The Heritage Foundation has 12 different levels of membership that come with incremental benefits as the donors give me. The standard donation form uses member language but does not detail what level their gift qualifies them for. During their annual membership renewal campaign, we wanted to find a way to let donors know their membership level based upon their giving amount while finding ways to upgrade them.

We devised a donation form that would display what level their current gift qualifies them form and what it would take to upgrade to the next level.  We made sure to include what benefits would come with the next level in membership.  Before rolling it out, we wanted to validate that it would improve the results of the campaign.

Research Question

Does adding language on the donation form that informs donors what member level they qualify for and then telling them the amount needed to move to the next level increase the average revenue per visitor compared to the standard donation form?


C: No Members Level Language
T1: Member Level Language


 Treatment NameConv. RateRelative DifferenceConfidenceAverage Gift
C: No Members Level Language 26.6%$56.12
T1: Member Level Language 17.4%-34.5% 99.2%$66.54

This experiment has a required sample size of 156 in order to be valid. Since the experiment had a total sample size of 569, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 34.5% decrease in conversion rate
× 18.6% increase in average gift
22.3% decrease in revenue

Key Learnings

The membership language treatment was able to lift the average gift of the donor by 18.6% but it negatively affected the conversion rate by 34.5%. This negative effect on conversion was enough to adversely effect the overall revenue of the form.

Upon digging deeper into the data, we discovered that the language would lift the average gift for donors above $50 but would cause lower level donors, those below $25, to not give at all. We may test membership language for future campaigns but limit the language so that only the larger donors see the language.

Experiment Documented by NextAfter

Question about experiment #447

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.