How an additional content offer checkbox affects conversion Experiment ID: #8986

Hoover Institution

Experiment Summary

Timeframe: 5/7/2018 - 5/11/2018

Hoover Institution was offering an eBook, Make America Exceptional Again to build their email file. They also had the Hoover Daily Report, a daily newsletter as an additional content offer. They wanted to give their visitors the chance to opt in to receive the daily newsletter in addition to the eBook, but didn’t want to disrupt conversion on the initial offer.

They decided to set up the checkbox opt-in on a separate page and split the traffic to test the impact of the second offer.

Research Question

Will adding an additional content offer through a checkbox decrease conversion?


C: first variant
T1: first variant copy 1


Treatment Name Conv. Rate Relative Difference Confidence
C: first variant 24.6%
T1: first variant copy 1 25.0% 1.6% 10.4%

This experiment has a required sample size of 95,144 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

The treatment produced no significant impact—increase or decrease—in conversion. Further data analysis showed that 73% of people who saw the treatment also opted into the Hoover Daily Report, which provided additional cultivation and engagement.

Ultimately, though there wasn’t a lift, this test was a win—conversion rate was not affected, and most users chose to receive additional content from Hoover Institution.

Experiment Documented by...

Jeff Giddens

Jeff is the President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.