How a radical redesign with simplified copy impacted a housefile digital appeal
Texas Public Policy Foundation
Experiment Summary
Ended On: 08/11/2021
As a part of the special legislative session, Texas Public Policy Foundation made an appeal around their proposed plan to lower property taxes.
The digital appeals to the housefile had been losing steam to convert donors in recent months, so they decided to test the language of the appeal and see what they could learn about the audience response.
The original email reflected a lot of policy-specific language and when graded in a readability tool, it was projected to be at a 12th grade reading level. So they developed a treatment that dramatically simplified the language and presented at a 5th grade reading level.
Research Question
We believe that simplifying the language for a housefile appeal email will achieve an increase in engagement.
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 23.4% | ||
| T1: | Treatment #1 | 36.2% | 54.5% | 100.0% |
This experiment has a required sample size of 99 in order to be valid. Since the experiment had a total sample size of 9,513, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
54.5% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
What we found was the simplified version of the email led to a 54.5% increase in the overall open rate on the email. Diving in deeper, we were not able to secure enough traffic to validate the clickthrough rate but the treatment had a 16% increase on clicks with a 52% level of confidence. This learning can certainly be replicated and applied for future digital appeals, but can also be tested in the acquisition space to see how prospective audiences respond to simplified language compared to the housefile. We can also identify a few donation pages where we may be able to test this simplified language concept.