How validating shared beliefs on a petition affects downstream donor conversion Experiment ID: #10692

Texas Public Policy Foundation

Experiment Summary

Ended On: 1/10/2019

Texas Public Policy Foundation was running a campaign to raise support for their Center for the American Future, which defended citizens against government overreach. They were running a petition of support for John Yearwood, a rancher who faced a threat from the government if he disturbed the habitat of a rare spider—one which he had never seen!

They ran an experiment on the petition page to see if adding a section that asked the reader to confirm their shared values—namely that Texas landowners should be able to use their land as they please, that the federal government should let property owners conserve their own land, and that the government should follow the 10th Amendment—would increase donor conversion. Previous testing had implied that these checkboxes could have an impact on email acquisition, but were more likely to have an impact on donations (more people would give after signing). They launched an A/B test to determine a winner. 

Research Question

How will validating shared beliefs on a petition affects downstream donor conversion

Design

C: Control
T1: Treatment #1

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Control 0.79%
T1: Treatment #1 0.38% -52.2% 96.9%

This experiment has a required sample size of 2,629 in order to be valid. Since the experiment had a total sample size of 6,351, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 52.2% decrease in conversion rate
× 0% increase in average gift

Key Learnings

While the checkboxes treatment had no effect on email acquisition, it cut downstream conversion by half. The hypothesis as to why is that “zooming out” to shared beliefs took the focus off of the story, which made the ask (which was hyper-relevant to Yearwood) less congruent with the landing page message. 


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Jeff Giddens

Jeff is the President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.