How simplifying an open letter’s readability impacted revenue Experiment ID: #20597

Americans for Prosperity

Experiment Summary

Timeframe: 4/15/2020 - 4/16/2020

As a part of the COVID-19 (Coronavirus) response efforts, American’s For Prosperity made available to concerned Americans an opportunity to add their name to an open letter to Congress as they planned a government response to provide citizens aid, as well as focused stimulus dollars to “restart the economy” again after the pandemic.

The open letter had become a wonderful engagement piece for the housefile, so they decided to open it up to prospective supporters to give them the opportunity to add their name alongside tens of thousands of other signers, as well.

Because the open letter was so popular, they decided to put a donation appeal behind it and test acquiring new donors, and did so very effectively.

But as time went on, the open letter signup (or email capture) rates were beginning to decline, so they wanted to experiment with increasing email signups. 

The original letter was carefully crafted by government affair and policy experts and when graded in a readability tool, it was projected to be at a 15th grade reading level. So, they thought: “What would happen if we dramatically simplified this language, and dropped it to a 5th grade reading level?

Research Question

Would simplifying the reading level of the open letter increase email signup, donor conversion or revenue?

Design

C: 15th Grade Reading Level
T1: 5th Grade Reading Level

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: 15th Grade Reading Level 0.53%
T1: 5th Grade Reading Level 2.2% 308.9% 99.4%

This experiment has a required sample size of 241 in order to be valid. Since the experiment had a total sample size of 3,272, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 308.9% increase in conversion rate
× 0% increase in average gift

Key Learnings

With a 99.4% level of confidence, we observed that the “dramatically simplified” readability of the open letter increased donor conversion rate by +309%.

Further, we observed that the average gift on the donations collected was +21% higher than those given through the control experience, which when combined with the increased donor conversion rate translated into an increase of revenue of +406% (LoC: 99.6%).

Finally, the treatment experience produced a 68% increase in email signup rate (LoC: 100%) over the control version, as well.

According to our readability analysis tool, the original letter was graded at a 15th grade reading level and took an estimated 2:35 seconds to read the letter, whereas the simplified treatment version of the letter was analyzed to be a 5th grade reading level and decreased the estimated time to read the letter to 1:00.

Subjectively, the treatment just appears more direct and better representative of the emotional mindset of a signer that is also ready to give and it took far less time for them to agree and sign their name.

Key Takeaway:

When it comes to pledges and petitions, simplicity is key. It is quite common that over-complicating (and really, over-manufacturing) the language that is used is ideal to give it “substance,” but this experiment proves that quick, decisive and concise emotional points and directives are more effective because (a) people are more likely to read it completely, and (b) they’re more likely to be motivated by it emotionally, which drives increased giving thereafter.


Experiment Documented by...

Greg Colunga

Greg is Executive Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.