How radically redesigning an email affected donation revenue
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Timeframe: 11/15/2017 - 11/20/2017
The Heritage Foundation created a popular offer for email acquisition with their Digital Guide to the Constitution. After a visitor downloaded the Digital Guide, they were offered membership to Heritage, along with an offer to get the print copy of the Guide to the Constitution for a gift of $100. This offer flow had been quite effective at converting prospects into donors using a text-based approach that had tested well with the house file. But since acquisition mailings reached an audience of people who were not as familiar with Heritage, they created a treatment to test that increased the force of the value proposition and introduced images of the offer.
The control, which introduced the free offer with a casual greeting, focused on the content contained in the Guide. The treatment created an enemy: “Right now, liberals are waging an all-out war against our nation’s founding principles…”. Instead of a resource, it offered a “gift”, and highlighted the hard work that went into its creation, before revealing that it was created for free. The treatment email also included an image of the offer to provide clarity.
Then, Heritage ran an A/B test to determine a winner.
Research Question
Will a radical redesign of the email that increases the force of the value proposition and introduces images increase response and revenue?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.01% | ||
| T1: | Treatment 1 | 0.02% | 123.1% | 98.6% |
This experiment has a required sample size of 93,509 in order to be valid. Since the experiment had a total sample size of 296,782, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 123.1% increase in conversion rate
× 0% increase in average gift
Key Learnings
The treatment produced a 123% increase in donations as well as an increase in revenue, which produced a big difference in return on investment. This showed the power of the increased value proposition, with the additional impact of the imagery to help the recipient visualize the offer.
This is a promising result that should be tested on more lists to ensure that there isn’t an anomaly with the first list choice.