How does longer form email copy affects clickthrough rate and donation conversion rate?
Colson Center
Experiment Summary
Timeframe: 12/16/2010 - 12/16/2010
Like many nonprofit organizations, the Colson Center had adapted their email strategy from their direct mail strategy. They sent very long letters from the organization’s founder in mail, and so when they launched their email appeal program, they took the very same approach. But are email and direct mail one in the same? In this experiment, the Colson Center challenged this paradigm and generated a new treatment that used dramatically less copy than the control. Once the email recipient clicked to the landing page, they would receive essentially the same copy as the longform email version, but would be one step closer to giving. The hypothesis is this: by lowering the perceived friction (length of copy) in the email, can we increase visits to the donation landing page and thereby increase giving?
Research Question
Which email will generated the highest clickthrough rate and ultimately the most donations?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control - Longform | 0.99% | ||
| T1: | Treatment 1 - Shortform | 13.0% | 1,209.2% | 100.0% |
This experiment has a required sample size of 28 in order to be valid. Since the experiment had a total sample size of 79,957, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
1,209.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The results of this experiment are very interesting because while clickthrough increased dramatically by over 1,200%, the increase in traffic did not significantly affect donations. Through other experiments we have learned that when it comes to email optimization, we have two dials that we can move that are codependent: Traffic and Conversion Rate. For example, we can increase Traffic by removing friction from the email, but that also means that the traffic we send is going to be much less motivated to take action and thereby producing a much lower Conversion Rate on the landing page. If the goal is to increase the total net donations or donation revenue, that sacrifice in Conversion Rate may be okay as long as the increase in Traffic Volume yield additional net donations and revenue. In this particular case, that was not the results so we concluded that we needed to test this same concept again during the next appeal month.
View the Results of the retest of the “Longform” vs. “Shortform” experiment here.