How simplifying petition language impacted revenue
Canadian Taxpayers Federation
A not-for-profit citizen's advocacy group dedicated to lower taxes, less waste and accountable government.
Experiment Summary
Ended On: 04/23/2020
The Canadian Taxpayers Federation (CTF) is a federally incorporated, not-for-profit citizen’s group dedicated to lower taxes, less waste and accountable government.
As a part of their efforts to urge the Canadian federal government in their efforts to provide relief to Canadian citizens in response to COVID-19 (coronavirus), they put together an open letter campaign that allowed Canadian citizens to urge their lawmakers for fiscal responsibility in response to the pandemic.
The open letter was followed by an instant donation page which provided open letter signers to make a gift to help reach more people to sign the open letter.
Research Question
Would simplifying the contents of the open letter (by decreasing the evaluated “reading grade level” from 10th grade down to a 6th grade reading level) impact letter signature rates, donor conversion or revenue?
Design
Results
| Treatment Name | Revenue per Visitor | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control - 10th Grade Reading Level | $0.23 | $41.10 | ||
| T1: | Treatment - 6th Grade Reading Level | $0.44 | 90.8% | 95.3% | $65.60 |
This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was met so these experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 19.5% increase in conversion rate
× 59.6% increase in average gift
90.8% increase in revenue
Key Learnings
With 95.2% level of confidence, we observed a +90.8% increase in revenue for open letter signers who became donors in the subsequent instant donation page.
This increase was driven by two metrics, specifically:
- An increase in donor conversion rate for treatment signers by +22.5% (although the LoC was only 66.5% and did not validate), and
- The average gift for donors in the treatment experience was +59.6% higher than the control donors.
Ironically, the email signup rate was rather unaffected by the treatment (improving only 0.7%, with a LoC of 25.8%).
Key Takeaways:
When you read the two treatments, the simplification of the letter content didn’t really “take anything away” from the content—in fact, the simplification made the letter appear more direct to the lawmaker it was being sent to. Our theory is that this increased “directness” of the message was a better representation of the reader/signer’s message, and when presented with an ask, they gave more generously on the instant donation page.
Because the organization was looking to maximize return on investment, the treatment was rolled out to 100% of the audience. Further experimentation may be required to verify if this approach could work again in the future.