How a long-form, donor-centered appeal decreased donor conversion rates on the primary donate page Experiment ID: #12712

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Timeframe: 3/8/2019 - 3/13/2019

We wanted to test a donor-centered message on the primary donation page, as every treatment historically has been focused on an organizationally centered series of value propositions.

Research Question

Does a donor centered message help improve donor conversion rate (or revenue) on the primary donation form for Heritage?


C: Control - Original
T1: Treatment - Donor Centered Message


Treatment Name Conv. Rate Relative Difference Confidence Average Gift
C: Control - Original 15.7% $0.00
T1: Treatment - Donor Centered Message 9.7% -37.8% 94.8% $0.00

This experiment has a required sample size of 242 in order to be valid. Since the experiment had a total sample size of 472, and the level of confidence is not above 95% the experiment results are not valid.

Key Learnings

With a 94.7% level of confidence, we validated that the treatment decreased the donor conversion rate by -37.8%.

Although the donor conversion rate decreased in the treatment segment, because the average gift in the treatment jumped pretty substantially (+61.3%) compared to the control, the revenue difference between the segments was only -0.6% (with a 50.86% LoC).

Our hypothesis is that the copy length led to the decrease in donor conversion rate, but we were encouraged by the increase in average gift from the treatment, so we will now develop and launch a short-form version of a donor-centered message to experiment against the short-form control message.

Experiment Documented by...

Greg Colunga

Greg is Executive Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.