The Heritage Foundation

How asking an open-loop question on an ad image affects click through rates

Experiment ID: #96562

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Timeframe: 04/25/2022 - 06/06/2022

The Heritage Foundation wanted to improve CTR on their Big Tech Survey campaign. The original ad had long, issue-focused copy and a statement on the ad image telling the audience to take their survey to fight back against big tech.

in an attempt to earn more clicks, Heritage tested two ad treatments: one with the same image/statement but shorter body copy and a second with a new image and an open loop question to encourage people to click to find out what they knew.

Research Question

We believe that asking an open loop question on the ad image for Facebook users will achieve increase donation rates.

Design

C: Control
T1: Treatment #1
T2: Treatment #2

Results

  Treatment Name Click Rate Relative Difference Confidence
C: Control 0.52%
T1: Treatment #1 0.51% -0.8% 5.1%
T2: Treatment #2 2.1% 307.8% 100.0%

This experiment has a required sample size of 766 in order to be valid. Since the experiment had a total sample size of 195,623, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    307.8% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

After running the test for 42 days, Heritage observed that treatment #1 (with shorter body copy but same statement on ad image) produced a 3.5% lift but with only a 23.1% LoC.

However, treatment #2 (with open loop question on ad image) produced a dramatic lift of 304% with a 100% LoC. This indicates that by opening a loop of curiosity on the most prominent part of the ad, Heritage was able to inspire more people to click through, presumably because they wanted to test their knowledge — confirming the hypothesis that creating curiosity would result in a lift in clicks.

This test also supports the general supposition that making changes to the ad’s most prominent element – the ad image – may have a greater impact than making similar changes to the body copy, which is less prominent.

Next, Heritage may choose to test a single element of this concept — either the ad image or the value prop language — to see which element has a bigger impact when changed. But for now, we can conclude that:

  1. Making changes to the ad image has a larger impact on performance than does changing the body copy
  2. Opening a curiosity loop is likely to encourage more people to click through


Experiment Documented by NextAfter

Question about experiment #96562

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.