How including the details of the plan decreased new donor revenue Experiment ID: #21497

Americans for Prosperity

Experiment Summary

Ended On: 6/23/2020

As a part of the Coronavirus (COVID-19) response efforts for Americans for Prosperity, they were running an acquisition offer which presented an Open Letter (essentially a petition) to Congress to not send bail out money to fiscally irresponsible states as a part of the recovery efforts.

In reviewing the offer donation page, we decided to test whether or not including the details of the plan to use grassroots activists to stop this from happening would increase donor conversion rates and/or revenue.

Research Question

Would adding the details of your plan increase revenue?

Design

C: Control — Repeating the Open Letter Statements
T1: Treatment — Detailing the plan to stop the bill

Results

Treatment Name Revenue per Visitor Relative Difference Confidence Average Gift
C: Control — Repeating the Open Letter Statements $0.64 $22.14
T1: Treatment — Detailing the plan to stop the bill $0.27 -56.9% 98.5% $24.38

This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was met so these experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 60.8% decrease in conversion rate
× 10.1% increase in average gift
56.9% decrease in revenue

Key Learnings

With a 98.4% level of confidence, we observed a decrease in revenue by -56.9%.

This was largely driven by the decrease in donor conversion rate of 60.8% (LoC: 98.3%), which also validated as a part of this experiment.

It’s important to note that this donation page was used as a part of an acquisition campaign, so the findings could be isolated just to newly acquired first time donors who are giving to this effort after signing the open letter/petition. Further testing with the housefile to include plan details as a part of your donation page(s) might be important to know over time.

However, it at least appears that the primary motivation for donors after signing an open letter is to (a) strengthen their message by adding more petition signers, and (b) they don’t need to see the details of your plans moving forward to make the outcome they’re pursuing become a reality.


Experiment Documented by...

Greg Colunga

Greg is Executive Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.