Harvest Ministries

How increasing clarity in email copy affects clickthrough rate

Experiment ID: #2944

Harvest Ministries

Harvest Christian Fellowship exists to bring Christians closer to God and to bring nonbelievers to a saving relationship with Him by showing how God's Word and faith in Him are applicable and relevant to everyday life.

Experiment Summary

Timeframe: 04/01/2016 - 05/02/2016

Harvest Ministries was sending a mid-month update on their fundraising appeal for the month. The offer was a book by Joni Earackson Tada. The control email highly incentivized the click by asking the reader to “find out why” the sender (Greg Laurie) admires Joni, the author of the book. While this might drive a lot of clicks, we’ve seen this approach drive much lower conversion rates, as those who click through are not as motivated to accept the offer.

We devised a treatment that moved some of that copy into the email, incentivizing the click by  giving them a timely call-to-action to find out how to get their own copy of the book before Mother’s Day.

We split the file and ran a test to see which email won.

Research Question

Will copy with a clearer call-to-action increase clickthrough rate?

Design

C: Find out why
T1: Get your copy

Results

 Treatment NameClick RateRelative DifferenceConfidence
C: Find out why 2.2%
T1: Get your copy 0.89%-58.7% 100.0%

This experiment has a required sample size of 718 in order to be valid. Since the experiment had a total sample size of 81,181, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    58.7% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The treatment produced a 58.7% decrease in clickthrough rate, which tells us that tipping our hand at the book offer reduced motivation to click through. However, the treatment produced a 55.1% lift in donor conversion on the following page, which tells us that those who did click through were more motivated to accept the offer.

Because of the disparity in clickthrough rate, the control conversion path produced more overall revenue, despite the lower conversion rate.


Experiment Documented by Jeff Giddens
Jeff Giddens is President of NextAfter.

Question about experiment #2944

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.