How credibility affects conversion Experiment ID: #106

Colson Center

Experiment Summary

Timeframe: 1/27/2011 - 2/1/2011

This experiment is based on a previous experiment that was conducted a few months before.  What we were trying to understand is how reducing friction (represented by length of copy) in an email affects clickthrough rate and ultimately donation response rate.  The previous experiment taught us that the shorter version of the email produced a much greater clickthrough rate, but did not significantly affect the donation response rate.  This experiment is essentially a retest of the original hypothesis that sought to determine what would ultimately generate the most gifts– reducing friction in the email or communicating greater value in the email.  Our thinking was that if we can get more people out of the email (where they can’t give a gift) to the landing page (where they can give a gift), we could increase conversions. We developed a treatment email that was shorter and to the point, with a highly incentivized call-to-action

Research Question

Would a shorter email that introduced the call-to-action quicker decrease friction and improve overall conversion rate compared to the longer email?


C: Long Email + Short Landing Page
T1: Short Email + Long Landing Page


Treatment Name Conv. Rate Relative Difference Confidence
C: Long Email + Short Landing Page 0.92%
T1: Short Email + Long Landing Page 0.64% -30.3% 100.0%

This experiment has a required sample size of 7,928 in order to be valid. Since the experiment had a total sample size of 77,645, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 30.3% decrease in conversion rate
× 0% increase in average gift

Key Learnings

Even though the treatment had a relative increase in clickthrough rate of 117% the shorter email treatment reduced conversion rate by 30.3%. After examining the treatment and control, we learned three critically important lessons about testing and optimization.  The first two have universal application to anyone who is engaged in testing and optimization and the final point lead to a breakthrough with the Colson Center:

  1. Best practices do not exist.  We had learned through testing with other research partners and from reading best practice guides that “shorter is better than longer.”  Some of our earlier experiments validated that and thought that was a universal law of email that applied to every organization.  However, with the Colson Center, this proved not to be true.
  2. When testing, make sure that you are validating the ultimate conversion goal.  Many organizations do not have adequate tracking set up to be able to validate the ultimate results of an email campaign and instead make decisions based on the “micro-goals” of Open Rates and Clickthrough Rates.  But if we would have made a decision on which email was best based on clickthrough alone, we would have really been shooting ourselves in the foot.  The ultimate measure of success must be validated based on the ultimate conversion goal, so tracking those goals all the way through the process is essential.
  3. In the case of the Colson Center, the messenger is more important than the message.  In the case of both the control and treatment version of the experiment, the exact same copy was used, except it was broken up differently.  In the Control (longform email, shortform landing page), most of the value proposition for giving a gift was being delivered in the email signed by Chuck Colson.  In the Treatment version (shortform email, longform landing page), the value proposition was delivered by the landing page, which is signed by no one.  This lead to a remarkable breakthrough: when Chuck Colson delivers the ask, it carries with it much greater weight and sends highly motivated visitors to the landing page.  Even though we can get more than double the number of visitors to the page by lowering friction in the form of shorter copy, those visitors are much less motivated to take action.  Once we discovered this, it completely transformed all of our future appeals because we unlocked a critically important donor learning.

And that’s what testing and optimization is all about.  It is about turning the web into a living laboratory so that you can decode critical learnings about your donors that can unlock breakthroughs in your fundraising program.


Get the Detailed Case Study

Get a copy of the slides that highlight the different elements of the experiment, the variables that were changed, and the key takeaways and learnings that you can apply to your own organization.

Pick your format of choice:

Experiment Documented by...

Tim Kachuriak

Tim is the Chief Innovation & Optimization Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.