The Heritage Foundation

How donation page copy impacted donor conversion in a high urgency campaign

Experiment ID: #55345

The Heritage Foundation

Experiment Summary

Timeframe: 03/13/2021 - 04/03/2021

When reviewing the donation page for a high urgency renewal campaign, we wanted to test reducing brand references and making the copy more donor-focused. Since visitors to this page were returning members, our hypothesis was they already knew or at least had some familiarity with who The Heritage Foundation is and what they do. For the treatment, we reframed the copy to show instead what the membership gift would accomplish and highlight the need, rather than emphasizing what Heritage was already doing.

Research Question

We believe that focusing value proposition on the need for existing members will achieve an increase in conversion rate.

Design

C: Control

T1: Variant 1

Results

	Treatment Name	Conv. Rate	Relative Difference	Confidence	Average Gift
C:	Control	36.0%			$0.00
T1:	Variant 1	33.6%	-6.8%	95.1%	$0.00

This experiment has a required sample size of 2,896 in order to be valid. Since the experiment had a total sample size of 5,847, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

0% increase in traffic
× 6.8% decrease in conversion rate
× 0% increase in average gift

Key Learnings

One element that was removed from the treatment copy was social proof, which we know is a key factor in donor motivation. The control shares that The Heritage Foundation shares over 500,000 supporters. For future testing, we will look for opportunities to keep elements of social proof while reducing brand mentions to see if there is any notable effect. The control copy also had congruency in the headline and the final ask on the page, making it abundantly clear what the desired action on the page was while the treatment headline leaves some ambiguity. "Defend conservative change in America" - but how? There are multiple ways to do so. In future headline tests, we can draw the connection between renewing membership and defending conservative change in America so the reader is not left guessing. We'd also like to re-test this concept at a later date considering it is an annual campaign and donors may already be motivated to give during this season. Donation page visitors could be inclined to renew their membership regardless of reading through the value proposition on the page, especially if they have been a long-time donor.

Notable changes to other metrics due to the experiment:

Revenue experienced a 7.2% decrease with a 88% level of confidence.

Observations of the impact on other visitor segments in the experiment:

Desktop visitors had a 7.3% decrease in donations with a 94% level of confidence.
New visitors had a 9.0% decrease in donations with a 97% level of confidence.
Housefile Email visitors had a 6.3% decrease in donations with a 94% level of confidence.
18-24 visitors had a 45.7% increase in donations with an 89% level of confidence. (We'd like to do some extra digging into this directional lift for this younger age segment!)
45-54 visitors had a 29.7% decrease in donations with a 98% level of confidence.
65+ visitors had a 12.6% decrease in donations with a 93% level of confidence.

Experiment Documented by Rebekah Josefy

Rebekah Josefy is an Optimization Director at NextAfter.

Question about experiment #55345

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.