How the presentation of a video impacts engagement
Alliance Defending Freedom
Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.
Experiment Summary
Timeframe: 02/07/2018 - 02/21/2018
Alliance Defending Freedom had recently learned that they were going to be arguing a case in front of the Supreme Court. The implications of the case would be far reaching and the complexity of it required a little explanation. They wanted a simple way to communicate these details to the general public so they created a dedicated landing page with a video explaining it all.
With the goal of the landing page being to get people to watch and consume the video, we wanted to find the ideal way to do this. We decided to run a test to validate the two unique hypotheses:
- Does making the video look more like a YouTube video increase engagement?
- Is the copy getting in the way of the video by pushing it further down that page (causing people the abandon the page before finding it)?
We decided to create three different treatments that would allow us to measure each of these hypotheses both together and individually.
Research Question
Which variant will result in the greatest engagement with the video?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 1.9% | ||
| T1: | No Text and Youtube Video Image | 5.9% | 214.3% | 100.0% |
| T2: | YouTube Image | 5.3% | 181.1% | 100.0% |
| T3: | No Text | 3.2% | 69.2% | 97.5% |
This experiment has a required sample size of 370 in order to be valid. Since the experiment had a total sample size of 5,842, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 214.3% increase in conversion rate
× 0% increase in average gift
Key Learnings
All three treatments validated for a lift in engagement, however, it would appear that making the video look more like a normal YouTube video had the biggest impact on visitor engagement. Our hypothesis is that the YouTube design added clarity to the available actions on the page in a way that the custom video image did not. By shaping the page to be more like what visitors are use to seeing, we were able to increase their engagement.