How a less designed, text heavy email affects donor conversion
KUOW
Experiment Summary
Ended On: 05/01/2020
KUOW Public Radio is Seattle’s NPW news station. Their independent, nonprofit newsroom produces award-winning stories, podcasts and events. For a time sensitive fundraising campaign, KUOW decided to test their usual email design against a more simple, stripped down, less designed version to see if it would lead to more donations.
Research Question
Will a less designed email lead to more giving?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control - Designed | 0.41% | ||
| T1: | Treatment - Plain | 0.53% | 28.8% | 96.5% |
This experiment has a required sample size of 25,388 in order to be valid. Since the experiment had a total sample size of 58,985, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 28.8% increase in conversion rate
× 0% increase in average gift
Key Learnings
The stripped down, plain text email led to a 28.8% increase in response rate which was the main goal/focus of the experiment and suggests that KUOW should consider this approach again in future email appeals.
A few other interesting findings:
- There was a -24% difference in average gift (mostly driven by two larger than normal donations) but is something worth monitoring in future experiments to ensure this approach leads not only to more donations but more overall revenue.
- The designed email was 28% better at generating clicks or getting people to the donation page (100% confidence) but produced fewer donations overall which means the click to conversion rate of the stripped down email was actually 79.4% greater at converting clicks to donations (100% confidence). This goes to show that often what gets clicks isn’t what gets donations or, as is the case here, actually gets fewer donations.
It seems clear that a stripped down, plain text approach can work for KUOW so now they can begin testing other elements within this strategy like the tone, raw links, referencing past giving, the use of images, length of copy, etc.