How a video trailer vs the first video lesson as a YouTube ad affected instant donor conversion rate
Hoover Institution
Experiment Summary
Ended On: 01/06/2023
While we were in our Google Ads account we discovered that we were running the 1st video lesson of the course instead of the course trailer as an ad. We had already seen good performance from the 1st lesson and saw this as an opportunity to test the course trailer against it and see which one performed better.
Research Question
We believe that adding a trailer for YouTube viewers will achieve an increase in donor conversion rate.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 1.6% | ||
| T1: | Treatment #1 | 1.2% | -25.9% | 81.2% |
This experiment has a required sample size of 6,381 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
We wanted to try this experiment in order to understand which type of content was more appealing to YouTube audiences and to better understand what types of videos work on YouTube.
While the conversion rate did decrease on the treatment that actually isn’t as big of a key metric as the Cost per Donor.
Cost per donor
C: First Lesson $140.47
T: Trailer $126.18
What’s not factored in is the CPC.
C: $2.17
T: $1.52
The 1 minute trailer gets a -$0.65 difference per click. That adds up. This is the reason why the control video has a higher conversion rate but also a higher acquisition cost.
Cpc’s aren’t equal (and sometimes not comparable) in an ad auction, even when all other settings are the same, ad rank influences CPC’s on the individual ad level.
The control does have a higher conversion rate and a higher cost per donor.
The treatment has a lower conversion rate and a lower cost per donor.
Because our main KPI is Cost per Donor, the treatment has a current directional advantage.