How validating shared beliefs on a petition affects downstream donor conversion
Texas Public Policy Foundation
Experiment Summary
Ended On: 01/10/2019
Texas Public Policy Foundation was running a campaign to raise support for their Center for the American Future, which defended citizens against government overreach. They were running a petition of support for John Yearwood, a rancher who faced a threat from the government if he disturbed the habitat of a rare spider—one which he had never seen!
They ran an experiment on the petition page to see if adding a section that asked the reader to confirm their shared values—namely that Texas landowners should be able to use their land as they please, that the federal government should let property owners conserve their own land, and that the government should follow the 10th Amendment—would increase donor conversion. Previous testing had implied that these checkboxes could have an impact on email acquisition, but were more likely to have an impact on donations (more people would give after signing). They launched an A/B test to determine a winner.
Research Question
How will validating shared beliefs on a petition affects downstream donor conversion
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.79% | ||
| T1: | Treatment #1 | 0.38% | -52.2% | 96.9% |
This experiment has a required sample size of 2,629 in order to be valid. Since the experiment had a total sample size of 6,351, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 52.2% decrease in conversion rate
× 0% increase in average gift
Key Learnings
While the checkboxes treatment had no effect on email acquisition, it cut downstream conversion by half. The hypothesis as to why is that “zooming out” to shared beliefs took the focus off of the story, which made the ask (which was hyper-relevant to Yearwood) less congruent with the landing page message.