The Heritage Foundation

How follow-up email appeals to direct mail appeals affect donor conversion

Experiment ID: #11713

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Timeframe: 01/18/2006 - 02/18/2006

The Heritage Foundation wanted to understand the potential impact of sending follow-up email appeals to donors on their file who received direct mail appeals. There were differing internal opinions about this—some thought that email appeals would cannibalize direct mail appeals, while others thought that it might enhance overall conversion rate.

They developed three unique segments and designed a testing protocol to understand conversion rate for each:

  1. Donors who would only receive direct mail appeals.
  2. Donors who would only receive the email followup appeals.
  3. Donors who would receive both direct mail appeals and email followup appeals.

They launched the appropriate communications to each segment and monitored response to understand how each strategy affected results.

Research Question

How will sending follow-up email appeals along with direct mail appeals affect donor conversion?


C: Direct Mail Only
T1: Email Only
T2: Email and Direct Mail


  Treatment Name Conv. Rate Relative Difference Confidence
C: Direct Mail Only 14.9%
T1: Email Only 1.4% -90.6% 100.0%
T2: Email and Direct Mail 23.9% 60.5% 100.0%

This experiment has a required sample size of 28 in order to be valid. Since the experiment had a total sample size of 14,909, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 90.6% decrease in conversion rate
× 0% increase in average gift

Key Learnings

The segment that received the direct mail appeals and the email appeals had a 60.5% higher conversion rate than direct mail appeals alone or email appeals alone. The increase was more than the simple addition of the two conversion rates as well, which showed that the combination of the appeals did in fact increase the overall likelihood to give.

There’s a lot of generally accepted wisdom about the power of multichannel donors, but putting it to the test and getting real data is always necessary. This experiment helped set the multichannel strategy for Heritage’s appeals going forward.

Experiment Documented by Jeff Giddens
Jeff Giddens is President of NextAfter.

Question about experiment #11713

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.