How deeper survey engagement affects donor conversion Experiment ID: #9471

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Ended On: 10/2/2018

The Heritage Foundation was running their Annual Member Survey, which asked members for their opinions on key policy initiatives and then gave them the opportunity to give. The initial treatment for the survey came from the direct mail piece, which was a very basic survey that just asked members to check boxes with the policy initiatives they thought were important. This might have worked in direct mail, but digitally, this seemed very underwhelming to the Heritage team.

They crafted a second survey that was much longer—asking multiple, nuanced questions for each policy area and leaving room for open-ended responses. They knew that this was risky—common wisdom says that many people don’t know what to write in open-ended responses—but they thought that deeper engagement would result in higher conversion.

Research Question

How will a longer survey that requires more time and attention from each respondent affect donor conversion?


C: Control
T1: Treatment #1


Treatment Name Conv. Rate Relative Difference Confidence
C: Control 1.3%
T1: Treatment #1 2.4% 81.9% 99.4%

This experiment has a required sample size of 1,287 in order to be valid. Since the experiment had a total sample size of 4,657, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 81.9% increase in conversion rate
× 0% increase in average gift

Key Learnings

Interestingly, the longer and more engaging survey produced a 9% increase in survey completions, which might have led to an incremental increase in donations just based on volume. But when the team looked at the analytics, the treatment survey (longer, more engaging) produced an 81.9% increase in donations. This indicates that the more a survey requires, the more invested the user might be. And deeper investment, in this case, led to increased response rates in terms of donor conversions.

This brought up one significant question—what if Heritage tried this in the mail?

Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.

Experiment Documented by...

Jeff Giddens

Jeff is the President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.