How adding membership level indicators affects membership renewal
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Timeframe: 01/07/2015 - 01/08/2015
The Heritage Foundation has 12 different levels of membership that come with incremental benefits as the donors give me. The standard donation form uses member language but does not detail what level their gift qualifies them for. During their annual membership renewal campaign, we wanted to find a way to let donors know their membership level based upon their giving amount while finding ways to upgrade them.
We devised a donation form that would display what level their current gift qualifies them form and what it would take to upgrade to the next level. We made sure to include what benefits would come with the next level in membership. Before rolling it out, we wanted to validate that it would improve the results of the campaign.
Research Question
Does adding language on the donation form that informs donors what member level they qualify for and then telling them the amount needed to move to the next level increase the average revenue per visitor compared to the standard donation form?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | No Members Level Language | 26.6% | $56.12 | ||
| T1: | Member Level Language | 17.4% | -34.5% | 99.2% | $66.54 |
This experiment has a required sample size of 156 in order to be valid. Since the experiment had a total sample size of 569, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 34.5% decrease in conversion rate
× 18.6% increase in average gift
22.3% decrease in revenue
Key Learnings
The membership language treatment was able to lift the average gift of the donor by 18.6% but it negatively affected the conversion rate by 34.5%. This negative effect on conversion was enough to adversely effect the overall revenue of the form.
Upon digging deeper into the data, we discovered that the language would lift the average gift for donors above $50 but would cause lower level donors, those below $25, to not give at all. We may test membership language for future campaigns but limit the language so that only the larger donors see the language.