How passive “advertising” impacts a subscribers likelihood to give
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Timeframe: 12/18/2016 - 01/07/2017
Subscribers that saw one of the donor-focused articles were 197% more likely to make a end of year contribution than those subscribers that did not. This would indicate that targeting donors across multiple “channels” of communication significantly impacts their willingness to give.
Just to make sure we did not see any bias in the makeup of the segments, we analyzed each group. Both segments were had an even distribution of existing Heritage donors and regular subscribers. Additionally, the article did not seem to have any impact on average gift when we removed outlier gifts.
Research Question
Will showing Daily Signal subscribers articles about Heritage’s philanthropic need increase the donors’ likelihood to give?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Visitors that didn't see articles | 1.9% | ||
| T1: | Visitors that saw an article | 5.5% | 196.6% | 100.0% |
This experiment has a required sample size of 115 in order to be valid. Since the experiment had a total sample size of 197,541, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 196.6% increase in conversion rate
× 0% increase in average gift
Key Learnings
At the end of the month of December, The Daily Signal produced six different articles that highlighted the impact donors have on the organization and illustrated the need for additional contributions. The idea behind this tactic was to remind visitors of the need for donor support so that when they were solicited by email and direct mail later in the month, they would be more likely to give a gift.
Given the nature of the test, we were not able to make it a normal A/B experiment. In an ideal world, people would have been equally likely to see the article as not see it but that really isn’t possible. However, we were able to isolate the subscribers that saw that page and compare them to the subscribers that visited The Daily Signal but did not view any of the articles. Based upon these segments, we could then analyze the impact these articles had on year end giving.
As a reference, the various donor-focused articles are listed below.
- http://dailysignal.com/2016/12/20/how-your-donation-to-the-daily-signal-makes-a-difference/
- http://dailysignal.com/2016/12/21/your-support-for-the-daily-signal-is-critical-and-these-21-comments-are-pretty-awesome-too/
- http://dailysignal.com/2016/12/26/the-daily-signal-is-having-an-impact-here-are-4-examples/
- http://dailysignal.com/2016/12/27/what-its-like-to-be-the-daily-signals-white-house-correspondent/
- http://dailysignal.com/2016/12/28/reporting-from-the-war-zone-why-conflict-journalism-matters/
- http://dailysignal.com/2016/12/29/live-from-washington-its-the-daily-signal/