How the email call-to-action affects clickthrough rate
Harvest Ministries
Harvest Christian Fellowship exists to bring Christians closer to God and to bring nonbelievers to a saving relationship with Him by showing how God's Word and faith in Him are applicable and relevant to everyday life.
Experiment Summary
Timeframe: 05/01/2016 - 05/08/2016
Harvest Ministries was sending their fundraising appeal for the month. The offer was the book Coming Home by James MacDonald. The offer book connected with the message of the email – prodigal children. The control email encouraged the click by asking the reader to watch the video about Greg’s son Jonathan who was a wayward child. Video can be a great engagement piece, but it also might impact the donor conversion rate in the end because the video may be distracting from the ultimate goal of giving. We wanted to see if we could produce a better if not equally compelling call-to-action to the video in the email.
The treatment was much more personal in nature and we removed the call-to-action to watch the video all together. Instead, the treatment email had Greg personally sharing the story of his son Jonathan and making a personal recommendation of the offer book which connected back to his son Jonathan’s story. The call-to-action on the treatment was to get the book that Greg recommended.
We split the file and ran a test to see which email won.
Research Question
How would a CTA to watch a video vs a personal recommendation affect click-through?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control: Watch video | 9.9% | ||
| T1: | Treatment: Personal recommendation | 4.2% | -58.2% | 100.0% |
This experiment has a required sample size of 151 in order to be valid. Since the experiment had a total sample size of 82,675, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
58.2% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
While a personal recommendation in a personal toned email can be very compelling, the CTA to watch a video proved to be more compelling to readers. The treatment saw a 58.2% decrease in clicks. The control set up the story and video of Greg’s son nicely and readers wanted to hear more of it. Readers are more motivated to click and watch a video than to get a resource personal recommended by Greg.
While this test proved that video improves click-through, another test for this same email proved that the video was a distracting element to the ultimate end goal of giving.
But clickthrough rate isn’t everything — if we stopped measuring our funnel here, we would be in trouble! Though many more people clicked through when asked to watch a video, they didn’t convert as donors. They simply went to the landing page to watch the video. On the treatment version of the email (which turned the video content into text), a smaller number of readers clicked through but were much more motivated to give. There wasn’t a video on the landing page that caused friction and donor conversion increased by 203%. See the landing page optimization experiment here.