How simplifying an open letter’s readability impacted revenue
Americans for Prosperity
Experiment Summary
Timeframe: 04/15/2020 - 04/16/2020
As a part of the COVID-19 (Coronavirus) response efforts, American’s For Prosperity made available to concerned Americans an opportunity to add their name to an open letter to Congress as they planned a government response to provide citizens aid, as well as focused stimulus dollars to “restart the economy” again after the pandemic.
The open letter had become a wonderful engagement piece for the housefile, so they decided to open it up to prospective supporters to give them the opportunity to add their name alongside tens of thousands of other signers, as well.
Because the open letter was so popular, they decided to put a donation appeal behind it and test acquiring new donors, and did so very effectively.
But as time went on, the open letter signup (or email capture) rates were beginning to decline, so they wanted to experiment with increasing email signups.
The original letter was carefully crafted by government affair and policy experts and when graded in a readability tool, it was projected to be at a 15th grade reading level. So, they thought: “What would happen if we dramatically simplified this language, and dropped it to a 5th grade reading level?“
Research Question
Would simplifying the reading level of the open letter increase email signup, donor conversion or revenue?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | 15th Grade Reading Level | 0.53% | ||
| T1: | 5th Grade Reading Level | 2.2% | 308.9% | 99.4% |
This experiment has a required sample size of 241 in order to be valid. Since the experiment had a total sample size of 3,272, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 308.9% increase in conversion rate
× 0% increase in average gift
Key Learnings
With a 99.4% level of confidence, we observed that the “dramatically simplified” readability of the open letter increased donor conversion rate by +309%.
Further, we observed that the average gift on the donations collected was +21% higher than those given through the control experience, which when combined with the increased donor conversion rate translated into an increase of revenue of +406% (LoC: 99.6%).
Finally, the treatment experience produced a 68% increase in email signup rate (LoC: 100%) over the control version, as well.
According to our readability analysis tool, the original letter was graded at a 15th grade reading level and took an estimated 2:35 seconds to read the letter, whereas the simplified treatment version of the letter was analyzed to be a 5th grade reading level and decreased the estimated time to read the letter to 1:00.
Subjectively, the treatment just appears more direct and better representative of the emotional mindset of a signer that is also ready to give and it took far less time for them to agree and sign their name.
Key Takeaway:
When it comes to pledges and petitions, simplicity is key. It is quite common that over-complicating (and really, over-manufacturing) the language that is used is ideal to give it “substance,” but this experiment proves that quick, decisive and concise emotional points and directives are more effective because (a) people are more likely to read it completely, and (b) they’re more likely to be motivated by it emotionally, which drives increased giving thereafter.