How going direct to a petition impacted email registration rates
Americans for Prosperity
Experiment Summary
Timeframe: 04/24/2019 - 05/22/2019
Americans for Prosperity launched a quiz that led to a petition, which led to an instant donation page. We found that although the quiz was acquiring emails, that people were dropping off at the point when we asked them to sign the petition, which meant that visitors to the instant donation page were really low. In an attempt to boost both email registration and instant donation pageviews, we wanted to experiment removing the quiz from the process and directly promoting the petition as the first step.
Research Question
Will a simplified funnel, which removes the first step (a quiz) from the funnel improve email registration rates?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Economic Quiz -- Freedom | 0.09% | ||
| T1: | Economic Petition -- Freedom | 0.20% | 127.7% | 100.0% |
This experiment has a required sample size of 8,486 in order to be valid. Since the experiment had a total sample size of 250,270, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 127.7% increase in conversion rate
× 0% increase in average gift
Key Learnings
With a 99.9% level of confidence, we were able to lift email acquisition rates by cutting the quiz from the process and directly promoting the petition by +127.7%.
Subsequently, we did see directional increases in both donor conversions (+151%, LoC: 54%) and revenue (+151%, Loc: 77%), but neither statistic validated through the experiment, since so few gifts ultimately were secured through the offer.
Although the original thought was that the quiz would build momentum towards petition signatures, which could crescendo into generating instant donors and dollars, in practice we found that a more simplified two-step approach was confirmed to produce more emails, and directionally more donors and dollars.