How image format affects clickthrough rate and ad quality ranking Experiment ID: #19454

Hoover Institution

Experiment Summary

Ended On: 1/7/2020

Hoover Institution was running ads to promote their new eBook, How Not To Teach American History. They were testing two different messaging approaches, but in setting up the ads, they wondered if the cover of the book actually sold itself, and didn’t need any text alongside it. Furthermore, they wondered if a tall, cropped image that took up more vertical space would receive more engagement.

They quickly cropped a picture and created a third variant that shared the same copy as the control, but had a different image size, and launched all three to see which got the best results.

Research Question

How will image format affect clickthrough rate and ad quality ranking?

Design

C: Control
T1: Treatment #1
T2: Treatment #2

Results

Treatment Name Click Rate Relative Difference Confidence
C: Control 1.2%
T1: Treatment #1 0.91% -27.2% 69.2%
T2: Treatment #2 2.0% 57.1% 98.6%

This experiment has a required sample size of 1,869 in order to be valid. Since the experiment had a total sample size of 30,322, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    57.1% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

When analyzing the results, Hoover’s team realized that all of the conversions in the campaign had gone to the ad with the tall image format. Looking into it, there seemed to be two factors at play: first, the tall image had a 57% higher clickthrough rate. But secondly, Facebook had only given the tall image an “above average” ad ranking, which helped it get a lower CPM, more impressions, and more clicks for less money. All of that traffic resulted in more conversions, even though it had the same copy as the control.

There appears to be a relationship between the ad image and Facebook’s algorithm that helped this ad format break through and deliver more conversions, which deserves further testing.


Experiment Documented by...

Jeff Giddens

Jeff is the President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.