How image format affects clickthrough rate and ad quality ranking
Hoover Institution
Experiment Summary
Ended On: 01/07/2020
Hoover Institution was running ads to promote their new eBook, How Not To Teach American History. They were testing two different messaging approaches, but in setting up the ads, they wondered if the cover of the book actually sold itself, and didn’t need any text alongside it. Furthermore, they wondered if a tall, cropped image that took up more vertical space would receive more engagement.
They quickly cropped a picture and created a third variant that shared the same copy as the control, but had a different image size, and launched all three to see which got the best results.
Research Question
How will image format affect clickthrough rate and ad quality ranking?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 1.2% | ||
| T1: | Treatment #1 | 0.91% | -27.2% | 69.2% |
| T2: | Treatment #2 | 2.0% | 57.1% | 98.6% |
This experiment has a required sample size of 1,869 in order to be valid. Since the experiment had a total sample size of 30,322, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
57.1% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
When analyzing the results, Hoover’s team realized that all of the conversions in the campaign had gone to the ad with the tall image format. Looking into it, there seemed to be two factors at play: first, the tall image had a 57% higher clickthrough rate. But secondly, Facebook had only given the tall image an “above average” ad ranking, which helped it get a lower CPM, more impressions, and more clicks for less money. All of that traffic resulted in more conversions, even though it had the same copy as the control.
There appears to be a relationship between the ad image and Facebook’s algorithm that helped this ad format break through and deliver more conversions, which deserves further testing.