How asking an open-loop question on an ad image affects click through rates
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Timeframe: 04/25/2022 - 06/06/2022
The Heritage Foundation wanted to improve CTR on their Big Tech Survey campaign. The original ad had long, issue-focused copy and a statement on the ad image telling the audience to take their survey to fight back against big tech.
in an attempt to earn more clicks, Heritage tested two ad treatments: one with the same image/statement but shorter body copy and a second with a new image and an open loop question to encourage people to click to find out what they knew.
Research Question
We believe that asking an open loop question on the ad image for Facebook users will achieve increase donation rates.
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.52% | ||
| T1: | Treatment #1 | 0.51% | -0.8% | 5.1% |
| T2: | Treatment #2 | 2.1% | 307.8% | 100.0% |
This experiment has a required sample size of 766 in order to be valid. Since the experiment had a total sample size of 195,623, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
307.8% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
After running the test for 30 days, Heritage observed that treatment #2 (with open loop question on ad image) produced a dramatic lift of 304% with a 100% LoC. This indicates that by opening a loop of curiosity on the most prominent part of the ad, Heritage was able to inspire more people to click through, presumably because they wanted to test their knowledge — confirming the hypothesis that creating curiosity would result in a lift in clicks.
This test also supports the general supposition that making changes to the ad’s most prominent element – the ad image – may have a greater impact than making similar changes to the body copy, which is less prominent.
Next, Heritage may choose to test a single element of this concept — either the ad image or the value prop language — to see which element has a bigger impact when changed. But for now, we can conclude that:
- Making changes to the ad image has a larger impact on performance than changing the body copy
- Opening a loop of curiosity is likely to encourage more people to click through