How tailoring your ask to the COVID-19 (Coronavirus) crisis impacted donor conversion rate Experiment ID: #20516
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Ended On: 4/1/2020
During The Heritage Foundation’s annual Membership Card Campaign, the team decided to experiment with an upcoming membership renewal campaign email by tailoring the content and the call to action specifically to the COVID-19 crisis tethered to their core value proposition for membership.
We wanted to see how mentioning the COVID-19 crisis would impact renewal and giving.
Would tailoring an appeal based upon the COVID-19 crisis improve donor conversion rate?
|Treatment Name||Conv. Rate||Relative Difference||Confidence|
This experiment has a required sample size of 9 in order to be valid. Since the experiment had a total sample size of 277, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 81.1% decrease in conversion rate
× 0% increase in average gift
With a 99.9% level of confidence, we observed that the donor conversion rate in the experiment was decreased by -81.1%.
When it comes to a membership renewal campaign, it certainly appears that tying your core value proposition to the COVID-19 crisis does not motivate donors to give (or members to renew their support).
In Heritage’s case, certainly providing policy research and recommendations to lawmakers as they developed government policies in response to the crisis did not work.
It is worth noting that Heritage also used a different sender for this experiment, and the clickthrough rate was impacted (as shown in the number of sessions), which may have had an impact to the results, as well.
It is left to be seen and further testing may be required to test direct fundraising appeals for this core value proposition in light of the crisis, but it would appear that acknowledging the COVID-19 (or Coronavirus) at all in this campaign was not effective. Subsequently, the control messaging was used for the remainder of the series of emails in the campaign.