The Heritage Foundation

How using a “yes/no” ad for an ebook offer affects click-through rate

Experiment ID: #160296

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Ended On: 08/29/2023

The Heritage Foundation is offering a free ebook about the Second Amendment. All of the metrics in the funnel we’re performing above benchmarks except for the click-through rate and cost per click of the ads. We wanted to increase these metrics in hopes that our downstream metrics would improve as well.

In the control, the image shows a picture of the ebook with the value proposition in the copy of the ad.

In order to try and decrease our cost per click and increase our click-through rate, we wanted to test a “yes/no” version of the ad where we ask a question in the image and provide button look-a-likes that say yes and no. The same value proposition stayed in the copy of the ad.

Research Question

We believe that using a yes/no image for this ad for potential ebook downloaders will achieve a higher click-through rate because the image is more engaging and interactive which causes more people to get to the landing page.


C: Control
T1: Treatment #1


  Treatment Name Click Rate Relative Difference Confidence
C: Control 1.2%
T1: Treatment #1 4.6% 270.8% 100.0%

This experiment has a required sample size of 158 in order to be valid. Since the experiment had a total sample size of 106,200, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    270.8% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The key learning from this experiment is that using a yes/no image in the ad increased the click-through rate and brought more people to the landing page. The treatment saw a significant 270.8% increase in clicks for all traffic compared to the control. This suggests that the interactive and engaging nature of the yes/no image captured the attention of the audience and enticed them to click on the ad.

Although we saw a significant increase in click-through rate, the treatment ads had a negative affect on downstream conversion. The treatment ads saw a 79.8% decrease in email sign up rate with a 100% level of confidence. We believe this is because the ad no longer provided clarity as to what the next step was for the user if they clicked the ad.

In future experimentation, we should continue to explore the use of interactive and engaging images in our ads. This could include asking questions, providing quizzes, or using other interactive elements to encourage user engagement. Additionally, it would be beneficial to test different variations of the yes/no image to further optimize the click-through rate and potentially lower the cost per click.

Experiment Documented by NextAfter

Question about experiment #160296

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.