Americans for Prosperity

How oversimplifying an open letter’s readability impacted revenue

Experiment ID: #21354

Americans for Prosperity

Experiment Summary

Timeframe: 4/15/2020 - 4/29/2020

As a part of the COVID-19 (Coronavirus) response efforts, American’s For Prosperity made available to concerned Americans an opportunity to add their name to an open letter to Congress as they planned a government response to provide citizens aid, as well as focused stimulus dollars to “restart the economy” again after the pandemic.

The open letter had become a wonderful engagement piece for the housefile, so they decided to open it up to prospective supporters to give them the opportunity to add their name alongside tens of thousands of other signers, as well.

Because the open letter was so popular, they decided to put a donation appeal behind it and test acquiring new donors, and did so very effectively.

But as time went on, the open letter signup (or email capture) rates were beginning to decline, so they wanted to experiment with increasing email signups. 

The original letter was carefully crafted by government affair and policy experts and when graded in a readability tool, it was projected to be at a 15th grade reading level. So, they thought: “What would happen if we dramatically simplified this language, and dropped it to a 5th grade reading level?

Research Question

Would simplifying the reading level of the open letter increase email signup, donor conversion or revenue?

Results

Treatment Name Revenue per Visitor Relative Difference Confidence
C: $0.18
T1: Treatment - 5th Grade Reading Level $0.13 -25.9% 96.8% $19.94

This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was met so these experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 20.1% decrease in conversion rate
× 7.2% decrease in average gift
25.9% decrease in revenue

Key Learnings

We observed that with a 96.8% level of confidence, that this “simplified readability” open letter decreased revenue by -25.9%.

This decrease in revenue was driven by two factors:

  1. A directionally valid 20% decrease in donor conversion rate (LoC: 90.8%), and
  2. A 7.2% lower average gift for the fewer number of donors that gave through the treatment experience.

It’s worth noting that email signup rate on the simplified open letter was flat (+0.1%, LoC: 4.2%), as well.

Our hypothesis is that the treatment was “too simplified”—that it did not slow the reader and get them more emotionally invested in the letter content, which is necessary to increase motivation to give on the instant donation page that sits behind the letter signing experience.

How do we know this? Because the readability application that we used measured how long it would take someone to read the letter. The control was estimated to take 2:35 to read entirely, whereas the treatment was estimated to be 1:00 flat. This would be a “time spent reading” decrease of 62%.

Where do we go from here? Further experimentation is required and we advise that testing something between the control and this 5th grade reading level may be advisable.


Experiment Documented by Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.

Question about experiment #6519

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.