How oversimplifying an open letter’s readability impacted revenue
Americans for Prosperity
Experiment Summary
Timeframe: 04/15/2020 - 04/29/2020
As a part of the COVID-19 (Coronavirus) response efforts, American’s For Prosperity made available to concerned Americans an opportunity to add their name to an open letter to Congress as they planned a government response to provide citizens aid, as well as focused stimulus dollars to “restart the economy” again after the pandemic.
The open letter had become a wonderful engagement piece for the housefile, so they decided to open it up to prospective supporters to give them the opportunity to add their name alongside tens of thousands of other signers, as well.
Because the open letter was so popular, they decided to put a donation appeal behind it and test acquiring new donors, and did so very effectively.
But as time went on, the open letter signup (or email capture) rates were beginning to decline, so they wanted to experiment with increasing email signups.
The original letter was carefully crafted by government affair and policy experts and when graded in a readability tool, it was projected to be at a 15th grade reading level. So, they thought: “What would happen if we dramatically simplified this language, and dropped it to a 5th grade reading level?“
Research Question
Would simplifying the reading level of the open letter increase email signup, donor conversion or revenue?
Design
Results
| Treatment Name | Revenue per Visitor | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control - 15th Grade Reading Level | $0.18 | $21.49 | ||
| T1: | Treatment - 5th Grade Reading Level | $0.13 | -25.9% | 96.8% | $19.94 |
This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was met so these experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 20.1% decrease in conversion rate
× 7.2% decrease in average gift
25.9% decrease in revenue
Key Learnings
We observed that with a 96.8% level of confidence, that this “simplified readability” open letter decreased revenue by -25.9%.
This decrease in revenue was driven by two factors:
- A directionally valid 20% decrease in donor conversion rate (LoC: 90.8%), and
- A 7.2% lower average gift for the fewer number of donors that gave through the treatment experience.
It’s worth noting that email signup rate on the simplified open letter was flat (+0.1%, LoC: 4.2%), as well.
Our hypothesis is that the treatment was “too simplified”—that it did not slow the reader and get them more emotionally invested in the letter content, which is necessary to increase motivation to give on the instant donation page that sits behind the letter signing experience.
How do we know this? Because the readability application that we used measured how long it would take someone to read the letter. The control was estimated to take 2:35 to read entirely, whereas the treatment was estimated to be 1:00 flat. This would be a “time spent reading” decrease of 62%.
Where do we go from here? Further experimentation is required and we advise that testing something between the control and this 5th grade reading level may be advisable.