How radically redesigning an email affected donation revenue Experiment ID: #7900

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Timeframe: 11/15/2017 - 11/20/2017

The Heritage Foundation created a popular offer for email acquisition with their Digital Guide to the Constitution. After a visitor downloaded the Digital Guide, they were offered membership to Heritage, along with an offer to get the print copy of the Guide to the Constitution for a gift of $100. This offer flow had been quite effective at converting prospects into donors using a text-based approach that had tested well with the house file. But since acquisition mailings reached an audience of people who were not as familiar with Heritage, they created a treatment to test that increased the force of the value proposition and introduced images of the offer.

The control, which introduced the free offer with a casual greeting, focused on the content contained in the Guide. The treatment created an enemy: “Right now, liberals are waging an all-out war against our nation’s founding principles…”. Instead of a resource, it offered a “gift”, and highlighted the hard work that went into its creation, before revealing that it was created for free. The treatment email also included an image of the offer to provide clarity.

Then, Heritage ran an A/B test to determine a winner.

Research Question

Will a radical redesign of the email that increases the force of the value proposition and introduces images increase response and revenue?


C: Control
T1: Treatment 1


Treatment Name Conv. Rate Relative Difference Confidence
C: Control 0.01%
T1: Treatment 1 0.02% 123.1% 98.6%

This experiment has a required sample size of 93,509 in order to be valid. Since the experiment had a total sample size of 296,782, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 123.1% increase in conversion rate
× 0% increase in average gift

Key Learnings

The treatment produced a 123% increase in donations as well as an increase in revenue, which produced a big difference in return on investment. This showed the power of the increased value proposition, with the additional impact of the imagery to help the recipient visualize the offer.

This is a promising result that should be tested on more lists to ensure that there isn’t an anomaly with the first list choice.

Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.

Experiment Documented by...

Jeff Giddens

Jeff is the President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.