How using a “yes/no” ad for an ebook offer affects click-through rate
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Ended On: 08/29/2023
The Heritage Foundation is offering a free ebook about the Second Amendment. All of the metrics in the funnel we’re performing above benchmarks except for the click-through rate and cost per click of the ads. We wanted to increase these metrics in hopes that our downstream metrics would improve as well.
In the control, the image shows a picture of the ebook with the value proposition in the copy of the ad.
In order to try and decrease our cost per click and increase our click-through rate, we wanted to test a “yes/no” version of the ad where we ask a question in the image and provide button look-a-likes that say yes and no. The same value proposition stayed in the copy of the ad.
Research Question
We believe that using a yes/no image for this ad for potential ebook downloaders will achieve a higher click-through rate because the image is more engaging and interactive which causes more people to get to the landing page.
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 1.2% | ||
| T1: | Treatment #1 | 4.6% | 270.8% | 100.0% |
This experiment has a required sample size of 158 in order to be valid. Since the experiment had a total sample size of 106,200, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
270.8% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The key learning from this experiment is that using a yes/no image in the ad increased the click-through rate and brought more people to the landing page. The treatment saw a significant 270.8% increase in clicks for all traffic compared to the control. This suggests that the interactive and engaging nature of the yes/no image captured the attention of the audience and enticed them to click on the ad.
Although we saw a significant increase in click-through rate, the treatment ads had a negative affect on downstream conversion. The treatment ads saw a 79.8% decrease in email sign up rate with a 100% level of confidence. We believe this is because the ad no longer provided clarity as to what the next step was for the user if they clicked the ad.
In future experimentation, we should continue to explore the use of interactive and engaging images in our ads. This could include asking questions, providing quizzes, or using other interactive elements to encourage user engagement. Additionally, it would be beneficial to test different variations of the yes/no image to further optimize the click-through rate and potentially lower the cost per click.