Hoover Institution

How adding new video creative to an existing offer affected donor conversion rate

Experiment ID: #141361

Hoover Institution

Experiment Summary

Ended On: 03/21/2023

At Hoover, we had recently been testing video ads with a voiceover with some success. Some of our older ads did not have a voiceover and the cost to acquire a donor with those ads were increasing.  We wanted to test a creative refresh (that included a voiceover) and relaunch the campaign to compare performance side by side.

Research Question

We believe that creating a new video for YouTube viewers will achieve a higher donor conversion rate.


C: Control
T1: Treatment #1


 Treatment NameConv. RateRelative DifferenceConfidence
C: Control 0.02%
T1: Treatment #1 0.08%349.7% 99.7%

This experiment has a required sample size of 8,600 in order to be valid. Since the experiment had a total sample size of 48,000, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 349.7% increase in conversion rate
× 0% increase in average gift

Key Learnings

This lowered the gross cost per donor from $152.90 to $49.10 while increasing donor volume by 3.5x.

Overall, these results demonstrate the importance of regularly updating creative assets in advertising campaigns to maintain donor engagement and ultimately – lengthening the lifetime of a current offer.

The key learning from this experiment is that creating a new video ad for YouTube viewers with a voiceover significantly improved the donor conversion rate by 349.7% compared to the old ad without a voiceover. This result was achieved with a 99.7% statistical level of confidence, confirming the significance of the findings.

In future experimentation, it is essential to prioritize a creative refresh and to consistently track and compare the performance of old and new ads. Additionally, it would be valuable to test different variations of a voiceover, such as tone and style, to find the most effective version.

Experiment Documented by Riley Young
Riley Landenberger is Audience Engagement Manager at NextAfter.

Question about experiment #141361

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.