How video on a donation page affects donor conversion Experiment ID: #3970
Harvest Christian Fellowship exists to bring Christians closer to God and to bring nonbelievers to a saving relationship with Him by showing how God's Word and faith in Him are applicable and relevant to everyday life.
Timeframe: 5/1/2016 - 5/8/2016
Harvest Ministries was sending a mid-month update on their fundraising appeal for the month. The offer was the book Coming Home by James MacDonald. The offer book connected with the message of the email – prodigal children. The control email encouraged the click by asking the reader to watch the video about Greg’s son Jonathan who was a wayward child. While video can be very compelling and a great engagement piece, it also might impact the donor conversion rate because the video may be distracting from giving and we may lose the reader after they’ve watched the video.
We created a treated that was much more personal in nature and removed the video all together from the email and landing page. The treatment email had Greg personally sharing the story of his son Jonathan. We then had Greg make a personal recommendation of the offer book in the email and landing page and connected it back to his son Jonathan’s story.
We split the file and ran a test to see which version won.
How would removing the video from the landing page affect donor conversion?
|Treatment Name||Conv. Rate||Relative Difference||Confidence|
|T1:||Treatment: Book offer||0.24%||203.0%||100.0%|
This experiment has a required sample size of 4,697 in order to be valid. Since the experiment had a total sample size of 82,675, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 203.0% increase in conversion rate
× 0% increase in average gift
Prior to this test, we ran a series of tests for the email that pushed people to this landing page. In those tests, we found that the treatment email (email without the video in it) produced a decrease in open rate and a lower clickthrough rate which attracted fewer visitors to this landing page. But, as you can see from this test, those visitors were much more motivated to make a donation. By removing the video on the landing page, we removed a significant element of friction and increased the force of the value proposition, increasing donor conversion by 203%.