How does longer form email copy affects clickthrough rate and donation conversion rate? Experiment ID: #2555

Colson Center

Experiment Summary

Timeframe: 12/16/2010 - 12/16/2010

Like many nonprofit organizations, the Colson Center had adapted their email strategy from their direct mail strategy.  They sent very long letters from the organization’s founder in mail, and so when they launched their email appeal program, they took the very same approach.  But are email and direct mail one in the same?   In this experiment, the Colson Center challenged this paradigm and generated a new treatment that used dramatically less copy than the control.  Once the email recipient clicked to the landing page, they would receive essentially the same copy as the longform email version, but would be one step closer to giving.  The hypothesis is this: by lowering the perceived friction (length of copy) in the email, can we increase visits to the donation landing page and thereby increase giving?

Research Question

Which email will generated the highest clickthrough rate and ultimately the most donations?


C: Control - Longform
T1: Treatment 1 - Shortform


Treatment Name Click Rate Relative Difference Confidence
C: Control - Longform 0.99%
T1: Treatment 1 - Shortform 13.0% 1,209.2% 100.0%

This experiment has a required sample size of 28 in order to be valid. Since the experiment had a total sample size of 79,957, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    1,209.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The results of this experiment are very interesting because while clickthrough increased dramatically by over 1,200%, the increase in traffic did not significantly affect donations.  Through other experiments we have learned that when it comes to email optimization, we have two dials that we can move that are codependent: Traffic and Conversion Rate.  For example, we can increase Traffic by removing friction from the email, but that also means that the traffic we send is going to be much less motivated to take action and thereby producing a much lower Conversion Rate on the landing page.  If the goal is to increase the total net donations or donation revenue, that sacrifice in Conversion Rate may be okay as long as the increase in Traffic Volume yield additional net donations and revenue.  In this particular case, that was not the results so we concluded that we needed to test this same concept again during the next appeal month.

View the Results of the retest of the “Longform” vs. “Shortform” experiment here.

Get the Detailed Case Study

Get a copy of the slides that highlight the different elements of the experiment, the variables that were changed, and the key takeaways and learnings that you can apply to your own organization.

Pick your format of choice:

Experiment Documented by...

Tim Kachuriak

Tim is the Chief Innovation & Optimization Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.