How replacing a video thumbnail with a raw link affected email click rate Experiment ID: #20634

Dallas Theological Seminary

The DTS mission is, “to glorify God by equipping godly servant-leaders for the proclamation of His Word and the building up of the body of Christ worldwide.” They strive to help men and women fulfill the Great Commission and the Great Commandment, or more simply: Teach Truth. Love Well.

Experiment Summary

Ended On: 4/17/2020

With DTS, previous experiments had found that simplifying the email communications tended to increase engagement. This was applicable from solicitation emails to cultivation newsletters. We decided to test this concept with a video as well. Past experiments with similar organizations had shown that video thumbnail images tended to help increase clicks for a video-focused email but, given the recent DTS results, we wanted to validate that specifically with the DTS audience.

Research Question

Will replacing a video thumbnail image with a raw link to the video affect email click rate?

Design

C: Video Thumbnail
T1: Raw Video Link

Results

Treatment Name Click Rate Relative Difference Confidence
C: Video Thumbnail 1.3%
T1: Raw Video Link 1.7% 27.0% 100.0%

This experiment has a required sample size of 8,866 in order to be valid. Since the experiment had a total sample size of 138,293, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    27.0% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

DTS has typically included thumbnail images in all emails containing video. Through this test, we were able to determine a 27% increase in engagement and traffic with the email that did not include the thumbnail image of the video. DTS should continue to use raw links in emails containing video moving forward.


Experiment Documented by...

Rebekah Josefy

Rebekah is Optimization Associate. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.