How Facebook ad imagery impacts clickthrough rate
Illinois Policy Institute
Experiment Summary
Timeframe: 10/23/2017 - 12/15/2017
Illinois Policy Institute recently discovered that showing a cartoon image in a Facebook ad decreased clickthrough rate. They decided to create two new treatments based to test against the control. The first treatment included an image of Michael Madigan–the state legislator pushing for an increase in taxes. The second treatment included an image of a house, to align with the subject of property taxes. They wondered: Which Facebook ad approach will have the highest clickthrough rate? They launched a 3-part test to find out.
Research Question
Which type of Facebook ad image will have the highest clickthrough rate?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 7.0% | ||
| T1: | Treatment 1 | 11.9% | 70.4% | 100.0% |
| T2: | Treatment 2 | 5.3% | -23.9% | 100.0% |
This experiment has a required sample size of 240 in order to be valid. Since the experiment had a total sample size of 195,902, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
70.4% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
Illinois Policy Institute found that the first treatment using Madigan’s image increased clickthrough rate by 70%, while the second treatment using the house image actually decreased ad clickthrough rate by 23%. This experiment is a good example of how an active A/B testing approach can lead to incremental performance when done consistently. Despite experiencing a drop in the second treatment, the learning in the first treatment outweighed the impact and will now be implemented as the winner.
It’s apparent that the use of Madigan’s image increased audience motivation. Many Illinoisans are familiar with this particular legislator. When connected with the topic of a potential property tax hike and their ability to take action to stop it, response rate went up significantly.