Hoover Institution

How a video trailer vs the first video lesson as a YouTube ad affected instant donor conversion rate

Experiment ID: #122541

Hoover Institution

Experiment Summary

Ended On: 01/06/2023

While we were in our Google Ads account we discovered that we were running the 1st video lesson of the course instead of the course trailer as an ad. We had already seen good performance from the 1st lesson and saw this as an opportunity to test the course trailer against it and see which one performed better.

Research Question

We believe that adding a trailer for YouTube viewers will achieve an increase in donor conversion rate.


C: Control
T1: Treatment #1


 Treatment NameConv. RateRelative DifferenceConfidence
C: Control 1.6%
T1: Treatment #1 1.2%-25.9% 81.2%

This experiment has a required sample size of 6,381 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

We wanted to try this experiment in order to understand which type of content was more appealing to YouTube audiences and to better understand what types of videos work on YouTube.

While the conversion rate did decrease on the treatment that actually isn’t as big of a key metric as the Cost per Donor.

Cost per donor

C: First Lesson $140.47

T: Trailer $126.18

What’s not factored in is the CPC.

C: $2.17

T: $1.52

The 1 minute trailer gets a -$0.65 difference per click. That adds up. This is the reason why the control video has a higher conversion rate but also a higher acquisition cost.

Cpc’s aren’t equal (and sometimes not comparable) in an ad auction, even when all other settings are the same, ad rank influences CPC’s on the individual ad level.

The control does have a higher conversion rate and a higher cost per donor.

The treatment has a lower conversion rate and a lower cost per donor.

Because our main KPI is Cost per Donor, the treatment has a current directional advantage.

Experiment Documented by Riley Young
Riley Landenberger is Audience Engagement Manager at NextAfter.

Question about experiment #122541

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.