How follow-up email appeals to direct mail appeals affect donor conversion
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Timeframe: 01/18/2006 - 02/18/2006
The Heritage Foundation wanted to understand the potential impact of sending follow-up email appeals to donors on their file who received direct mail appeals. There were differing internal opinions about this—some thought that email appeals would cannibalize direct mail appeals, while others thought that it might enhance overall conversion rate.
They developed three unique segments and designed a testing protocol to understand conversion rate for each:
- Donors who would only receive direct mail appeals.
- Donors who would only receive the email followup appeals.
- Donors who would receive both direct mail appeals and email followup appeals.
They launched the appropriate communications to each segment and monitored response to understand how each strategy affected results.
Research Question
How will sending follow-up email appeals along with direct mail appeals affect donor conversion?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Direct Mail Only | 14.9% | ||
| T1: | Email Only | 1.4% | -90.6% | 100.0% |
| T2: | Email and Direct Mail | 23.9% | 60.5% | 100.0% |
This experiment has a required sample size of 28 in order to be valid. Since the experiment had a total sample size of 14,909, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 90.6% decrease in conversion rate
× 0% increase in average gift
Key Learnings
The segment that received the direct mail appeals and the email appeals had a 60.5% higher conversion rate than direct mail appeals alone or email appeals alone. The increase was more than the simple addition of the two conversion rates as well, which showed that the combination of the appeals did in fact increase the overall likelihood to give.
There’s a lot of generally accepted wisdom about the power of multichannel donors, but putting it to the test and getting real data is always necessary. This experiment helped set the multichannel strategy for Heritage’s appeals going forward.