How including the details of the plan decreased new donor revenue
Americans for Prosperity
Experiment Summary
Ended On: 06/23/2020
As a part of the Coronavirus (COVID-19) response efforts for Americans for Prosperity, they were running an acquisition offer which presented an Open Letter (essentially a petition) to Congress to not send bail out money to fiscally irresponsible states as a part of the recovery efforts.
In reviewing the offer donation page, we decided to test whether or not including the details of the plan to use grassroots activists to stop this from happening would increase donor conversion rates and/or revenue.
Research Question
Would adding the details of your plan increase revenue?
Design
Results
| Treatment Name | Revenue per Visitor | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control — Repeating the Open Letter Statements | $0.64 | $22.14 | ||
| T1: | Treatment — Detailing the plan to stop the bill | $0.27 | -56.9% | 98.5% | $24.38 |
This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was met so these experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 60.8% decrease in conversion rate
× 10.1% increase in average gift
56.9% decrease in revenue
Key Learnings
With a 98.4% level of confidence, we observed a decrease in revenue by -56.9%.
This was largely driven by the decrease in donor conversion rate of 60.8% (LoC: 98.3%), which also validated as a part of this experiment.
It’s important to note that this donation page was used as a part of an acquisition campaign, so the findings could be isolated just to newly acquired first time donors who are giving to this effort after signing the open letter/petition. Further testing with the housefile to include plan details as a part of your donation page(s) might be important to know over time.
However, it at least appears that the primary motivation for donors after signing an open letter is to (a) strengthen their message by adding more petition signers, and (b) they don’t need to see the details of your plans moving forward to make the outcome they’re pursuing become a reality.