Alliance Defending Freedom

How the presentation of a video impacts engagement

Experiment ID: #3146

Alliance Defending Freedom

Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.

Experiment Summary

Timeframe: 02/07/2018 - 02/21/2018

Alliance Defending Freedom had recently learned that they were going to be arguing a case in front of the Supreme Court. The implications of the case would be far reaching and the complexity of it required a little explanation. They wanted a simple way to communicate these details to the general public so they created a dedicated landing page with a video explaining it all.

With the goal of the landing page being to get people to watch and consume the video, we wanted to find the ideal way to do this. We decided to run a test to validate the two unique hypotheses:

  1. Does making the video look more like a YouTube video increase engagement?
  2. Is the copy getting in the way of the video by pushing it further down that page (causing people the abandon the page before finding it)?

We decided to create three different treatments that would allow us to measure each of these hypotheses both together and individually.

Research Question

Which variant will result in the greatest engagement with the video?


C: Control
T1: No Text and Youtube Video Image
T2: YouTube Image
T3: No Text


 Treatment NameConv. RateRelative DifferenceConfidence
C: Control 1.9%
T1: No Text and Youtube Video Image 5.9%214.3% 100.0%
T2: YouTube Image 5.3%181.1% 100.0%
T3: No Text 3.2%69.2% 97.5%

This experiment has a required sample size of 370 in order to be valid. Since the experiment had a total sample size of 5,842, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 214.3% increase in conversion rate
× 0% increase in average gift

Key Learnings

All three treatments validated for a lift in engagement, however, it would appear that making the video look more like a normal YouTube video had the biggest impact on visitor engagement. Our hypothesis is that the YouTube design added clarity to the available actions on the page in a way that the custom video image did not. By shaping the page to be more like what visitors are use to seeing, we were able to increase their engagement.

Experiment Documented by NextAfter

Question about experiment #3146

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.