Dallas Theological Seminary

How video ads affect email acquisition rate for a free course

Experiment ID: #6239

Dallas Theological Seminary

The DTS mission is, “to glorify God by equipping godly servant-leaders for the proclamation of His Word and the building up of the body of Christ worldwide.” They strive to help men and women fulfill the Great Commission and the Great Commandment, or more simply: Teach Truth. Love Well.

Experiment Summary

Timeframe: 02/03/2017 - 03/03/2017

DTS was promoting their new course, called Can You Trust the Bible?. They had a special video from their Academic Dean to promote it and wanted to test using this video to promote the course. Video ads had shown in some experiments to improve conversion for offers, although DTS had previously used static ads.

They split audiences and tested each ad unit to see which resulted in a better cost-per-acquisition.

 

Research Question

Will video ads produce an increase in email acquisition rate?

Design

C: Static ad
T1: Video ad

Results

 Treatment NameConv. RateRelative DifferenceConfidence
C: Static ad 10.5%
T1: Video ad 6.6%-37.1% 99.5%

This experiment has a required sample size of 420 in order to be valid. Since the experiment had a total sample size of 1,633, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 37.1% decrease in conversion rate
× 0% increase in average gift

Key Learnings

The video ads produced a 37% decrease in conversion when tested against the static ads. While a deeper look at the data showed that many more people engaged with the video ads (which means they clicked somewhere on the ad), the video ad was either too distracting or not compelling enough to convince them to take the next step and sign up for the course. This seems to indicate that, in this case, video is more of a distraction than a motivator to take the course.

This doesn’t entirely rule out video ads on Facebook. There have been experiments where video ads produce a lift, and they can be effective at remarketing. DTS will continue to test video as a promotional tool for their free courses.


Experiment Documented by Jeff Giddens
Jeff Giddens is President of NextAfter.

Question about experiment #6239

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.