The Heritage Foundation

How a new landing page design impacted donor conversion

Experiment ID: #5641

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Timeframe: 10/17/2016 - 11/10/2016

The Heritage.org website is in the process of a redesign. As part of this transition, it was suggested that the existing donation page design be migrated to the new format. This would include the transition to a new look and feel for the donation form itself. Before making this change, we wanted to measure what kind of impact it would have on the performance of the donation pages.

Research Question

How will the new redesigned page and donation form impact donations?

Design

C: Original Template
T1: Heritage Redesign

Results

 Treatment NameConv. RateRelative DifferenceConfidenceAverage Gift
C: Original Template 6.1%$79.97
T1: Heritage Redesign 6.1%0.7% 13.2%$442.52

This experiment has a required sample size of 2,532,896 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

After running for the length of a money bomb campaign, we determined that the new design had no statistically significant impact on the performance of the campaign. While this is reassuring, it has led to two more questions:

  1. How does the donation form design specifically impact conversion? We want to isolate this factor to ensure the more modern design is not having a negative impact on the older donor base.
  2. Once the new site launches, will this new design actually increase conversion by maintaining visual congruence.

Both of these questions will be tested in upcoming campaigns.


Experiment Documented by NextAfter

Question about experiment #5641

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.