How length and presentation of email copy impact donor conversion rate
Leadership Institute
Experiment Summary
Timeframe: 04/10/2018 - 04/25/2018
Leadership Institute was sending their monthly appeal to their email file, based on the direct mail letter that also went out during that month. They were curious to see how including the full text (including extra spacing) would perform as compared to a truncated email version of the copy that distilled the value proposition into a shorter length email.
They created both versions and split the housefile to test.
Research Question
How will length and presentation of email copy impact conversion rate?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 27.5% | ||
| T1: | Treatment #1 | 11.6% | -57.7% | 99.6% |
This experiment has a required sample size of 42 in order to be valid. Since the experiment had a total sample size of 269, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 57.7% decrease in conversion rate
× 0% increase in average gift
Key Learnings
Though this experiment produced a big lift in the clickthrough rate, it resulted in a conversion rate that was half of the control—meaning that “more clicks” did not result in more donors. This shows the power of copy to motivate email readers to action—more copy produced fewer, more motivated clicks.
This suggests a future test to adjust the presentation of the email without shortening the copy.