How an animated video ad image increased ad clickthrough rate Experiment ID: #18708

The Fund for American Studies

Experiment Summary

Timeframe: 9/30/2019 - 10/14/2019

As a part of optimizing The Fund for American Studies’ eBook offer, we wanted to increase ad clickthrough rate to improve traffic through the funnel. To experiment with boosting ad clickthrough, we decided to

Research Question

Will using an animated video ad from Facebook increase ad clickthrough rate?

Design

C: Static Ad
T1: Video Ad

Results

Treatment Name Click Rate Relative Difference Confidence
C: Static Ad 1.7%
T1: Video Ad 1.9% 7.7% 81.1%

This experiment has a required sample size of 75,758 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

Directionally, we observed an increase of ad clickthrough rate by +7.7%, however we did not validate the experiment (achieving only an 81% level of confidence).

We turned off this experiment because after running for two weeks, we achieved the (directional) lift in ad clickthrough rate and traffic, however the email registration rate decreased (-28.5%, LoC: 100%), as did the donor conversion rate (-41.6%, LoC: 65.7%).

Our hypothesis is that although the video ad produced more traffic, with the decrease in email signups and donations, it was likely erroneous click activity after watching (or attempting to rewatch) the ad.

This approach has worked well for some offers and causes, and we should consider retesting this tactic at some point down the road.


Experiment Documented by...

Greg Colunga

Greg is Executive Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.