How emphasizing a single day of giving impacted overall conversion
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Timeframe: 04/10/2018 - 04/24/2018
For the 2018 Tax Day Moneybomb, Heritage decided to test out a radical concept that had been observed in the nonprofit community. The standard moneybomb campaign would run for 10+ days and would contain over 7 unique communications. In each communication, a direct ask would be made with a link to a donation page. These donors were asked to make a gift before a certain deadline with the emphasis on making the gift that day.
The treatment we wanted to test out was the concept of a one-day campaign. Instead of asking donors to make a gift prior to the end of a certain date, we asked them to give on a specific date. For those subscribers in the treatment group, every single communication in the campaign used this same tactic and even included additional elements like iCal attachments to help hold the date. In each email, we did include a link to a donation page if they wanted to make a gift immediately but the emphasis was put on donating on that one day.
Research Question
Will emphasizing a single day of giving increase overall donor conversion for the campaign?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Consistent Asks | 0.04% | ||
| T1: | One Day Emphasis | 0.03% | -29.2% | 99.8% |
This experiment has a required sample size of 192,226 in order to be valid. Since the experiment had a total sample size of 1,032,243, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 29.2% decrease in conversion rate
× 0% increase in average gift
Key Learnings
As a note, the images above are the initial emails for this campaign but all subsequent emails used the same tactic. We found that emphasizing this single day of giving actually decreased giving by 29%. There are two potential hypotheses that may explain these results:
- By providing the bail-out option to give a gift in each email, the treatment actually presented conflicting messages (“wait to give” and “give today”). This conflicting message may have resulted in donor confusion and, subsequently, lower conversion.
- The one-day emphasis campaign is not as effective as the long-term ask approach that we generally take with these campaigns.
We would need to run another campaign with the ability to only give on a single day to know which of these two hypotheses are true.