KUOW

Will a less designed, more text based email lead to more donations?

Experiment ID: #21354

KUOW

Experiment Summary

Ended On: 5/1/2020

KUOW Public Radio is Seattle’s NPW news station. Their independent, nonprofit newsroom produces award-winning stories, podcasts and events. For a time sensitive fundraising campaign, KUOW decided to test their usual email design against a more simple, stripped down, less designed version to see if it would lead to more donations.

Research Question

Will a less designed email lead to more giving?

Design

C: Control - Designed
T1: Treatment - Plain

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: 0.41%
T1: Treatment - Plain 0.53% 28.8% 96.5%

This experiment has a required sample size of 25,388 in order to be valid. Since the experiment had a total sample size of 58,985, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 28.8% increase in conversion rate
× 0% increase in average gift

Key Learnings

The stripped down, plain text email led to a 28.8% increase in response rate which was the main goal/focus of the experiment and suggests that KUOW should consider this approach again in future email appeals.

A few other interesting findings:

  • There was a -24% difference in average gift (mostly driven by two larger than normal donations) but is something worth monitoring in future experiments to ensure this approach leads not only to more donations but more overall revenue.
  • The designed email was 28% better at generating clicks or getting people to the donation page (100% confidence) but produced fewer donations overall which means the click to conversion rate of the stripped down email was actually 79.4% greater at converting clicks to donations (100% confidence). This goes to show that often what gets clicks isn’t what gets donations or, as is the case here, actually gets fewer donations.

It seems clear that a stripped down, plain text approach can work for KUOW so now they can begin testing other elements within this strategy like the tone, raw links, referencing past giving, the use of images, length of copy, etc.


Experiment Documented by Brady Josephson
Brady Josephson is Managing Director of the NextAfter Institute.

Question about experiment #6527

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.