Harvest Ministries

How video on a donation page affects donor conversion

Experiment ID: #3970

Harvest Ministries

Harvest Christian Fellowship exists to bring Christians closer to God and to bring nonbelievers to a saving relationship with Him by showing how God's Word and faith in Him are applicable and relevant to everyday life.

Experiment Summary

Timeframe: 05/01/2016 - 05/08/2016

Harvest Ministries was sending a mid-month update on their fundraising appeal for the month. The offer was the book Coming Home by James MacDonald. The offer book connected with the message of the email – prodigal children. The control email encouraged the click by asking the reader to watch the video about Greg’s son Jonathan who was a wayward child. While video can be very compelling and a great engagement piece, it also might impact the donor conversion rate because the video may be distracting from giving and we may lose the reader after they’ve watched the video.

We created a treated that was much more personal in nature and removed the video all together from the email and landing page. The treatment email had Greg personally sharing the story of his son Jonathan. We then had Greg make a personal recommendation of the offer book in the email and landing page and connected it back to his son Jonathan’s story.

We split the file and ran a test to see which version won.

Research Question

How would removing the video from the landing page affect donor conversion?

Design

C: Control: Video
T1: Treatment: Book offer

Results

 Treatment NameConv. RateRelative DifferenceConfidence
C: Control: Video 0.08%
T1: Treatment: Book offer 0.24%203.0% 100.0%

This experiment has a required sample size of 4,697 in order to be valid. Since the experiment had a total sample size of 82,675, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 203.0% increase in conversion rate
× 0% increase in average gift

Key Learnings

Prior to this test, we ran a series of tests for the email that pushed people to this landing page. In those tests, we found that the treatment email (email without the video in it) produced a decrease in open rate and a lower clickthrough rate which attracted fewer visitors to this landing page. But, as you can see from this test, those visitors were much more motivated to make a donation. By removing the video on the landing page, we removed a significant element of friction and increased the force of the value proposition, increasing donor conversion by 203%.


Experiment Documented by Courtney Gaines
Courtney Gaines is Vice President at NextAfter.

Question about experiment #3970

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.