The Heritage Foundation

How tailoring your ask to the COVID-19 (Coronavirus) crisis impacted donor conversion rate

Experiment ID: #29664

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Ended On: 04/01/2020

During The Heritage Foundation’s annual Membership Card Campaign, the team decided to experiment with an upcoming membership renewal campaign email by tailoring the content and the call to action specifically to the COVID-19 crisis tethered to their core value proposition for membership.

We wanted to see how mentioning the COVID-19 crisis would impact renewal and giving.

Research Question

Would tailoring an appeal based upon the COVID-19 crisis improve donor conversion rate?

Design

C: Control
T1: Treatment #1

Results

 Treatment NameConv. RateRelative DifferenceConfidence
C: Control 40.3%
T1: Treatment #1 7.6%-81.1% 100.0%

This experiment has a required sample size of 9 in order to be valid. Since the experiment had a total sample size of 277, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 81.1% decrease in conversion rate
× 0% increase in average gift

Key Learnings

With a 99.9% level of confidence, we observed that the donor conversion rate in the experiment was decreased by -81.1%.

When it comes to a membership renewal campaign, it certainly appears that tying your core value proposition to the COVID-19 crisis does not motivate donors to give (or members to renew their support). 

In Heritage’s case, certainly providing policy research and recommendations to lawmakers as they developed government policies in response to the crisis did not work. 

It is worth noting that Heritage also used a different sender for this experiment, and the clickthrough rate was impacted (as shown in the number of sessions), which may have had an impact to the results, as well.

It is left to be seen and further testing may be required to test direct fundraising appeals for this core value proposition in light of the crisis, but it would appear that acknowledging the COVID-19 (or Coronavirus) at all in this campaign was not effective. Subsequently, the control messaging was used for the remainder of the series of emails in the campaign.


Experiment Documented by Greg Colunga
Greg Colunga is Executive Vice President at NextAfter.

Question about experiment #29664

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.