How adding new video creative to an existing offer affected donor conversion rate
Hoover Institution
Experiment Summary
Ended On: 03/21/2023
At Hoover, we had recently been testing video ads with a voiceover with some success. Some of our older ads did not have a voiceover and the cost to acquire a donor with those ads were increasing. We wanted to test a creative refresh (that included a voiceover) and relaunch the campaign to compare performance side by side.
Research Question
We believe that creating a new video for YouTube viewers will achieve a higher donor conversion rate.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.02% | ||
| T1: | Treatment #1 | 0.08% | 349.7% | 99.7% |
This experiment has a required sample size of 8,600 in order to be valid. Since the experiment had a total sample size of 48,000, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 349.7% increase in conversion rate
× 0% increase in average gift
Key Learnings
This lowered the gross cost per donor from $152.90 to $49.10 while increasing donor volume by 3.5x.
Overall, these results demonstrate the importance of regularly updating creative assets in advertising campaigns to maintain donor engagement and ultimately – lengthening the lifetime of a current offer.
The key learning from this experiment is that creating a new video ad for YouTube viewers with a voiceover significantly improved the donor conversion rate by 349.7% compared to the old ad without a voiceover. This result was achieved with a 99.7% statistical level of confidence, confirming the significance of the findings.
In future experimentation, it is essential to prioritize a creative refresh and to consistently track and compare the performance of old and new ads. Additionally, it would be valuable to test different variations of a voiceover, such as tone and style, to find the most effective version.