How passive “advertising” impacts a subscribers likelihood to give Experiment ID: #5988

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Timeframe: 12/18/2016 - 1/7/2017

At the end of the month of December, The Daily Signal produced six different articles that highlighted the impact donors have on the organization and illustrated the need for additional contributions. The idea behind this tactic was to remind visitors of the need for donor support so that when they were solicited by email and direct mail later in the month, they would be more likely to give a gift.

Given the nature of the test, we were not able to make it a normal A/B experiment. In an ideal world, people would have been equally likely to see the article as not see it but that really isn’t possible. However, we were able to isolate the subscribers that saw that page and compare them to the subscribers that visited The Daily Signal but did not view any of the articles. Based upon these segments, we could then analyze the impact these articles had on year end giving.

As a reference, the various donor-focused articles are listed below.

Research Question

Will showing Daily Signal subscribers articles about Heritage’s philanthropic need increase the donors’ likelihood to give?

MECLABS Conversion Factors Targeted

C = 4m + 3v + 2( i - f) - 2a ©

Copyright 2015, MECLABS

Design

C: Visitors that didn't see articles
T1: Visitors that saw an article

Results

Treatment Name Conv. Rate Relative Difference Confidence Average Gift
C: Visitors that didn't see articles 1.9% $86.63
T1: Visitors that saw an article 5.5% 196.6% 100.0% $83.32

This experiment has a required sample size of 115 in order to be valid. Since the experiment had a total sample size of 197,541, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 196.6% increase in conversion rate
× 3.8% decrease in average gift
185.3% increase in revenue

Key Learnings

Subscribers that saw one of the donor-focused articles were 197% more likely to make a end of year contribution than those subscribers that did not. This would indicate that targeting donors across multiple “channels” of communication significantly impacts their willingness to give.

Just to make sure we did not see any bias in the makeup of the segments, we analyzed each group. Both segments were had an even distribution of existing Heritage donors and regular subscribers. Additionally, the article did not seem to have any impact on average gift when we removed outlier gifts.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Kevin Peters

Kevin is a Senior Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.