How an additional content offer checkbox affects conversion
Hoover Institution
Experiment Summary
Timeframe: 05/07/2018 - 05/11/2018
Hoover Institution was offering an eBook, Make America Exceptional Again to build their email file. They also had the Hoover Daily Report, a daily newsletter as an additional content offer. They wanted to give their visitors the chance to opt in to receive the daily newsletter in addition to the eBook, but didn’t want to disrupt conversion on the initial offer.
They decided to set up the checkbox opt-in on a separate page and split the traffic to test the impact of the second offer.
Research Question
Will adding an additional content offer through a checkbox decrease conversion?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | first variant | 24.6% | ||
| T1: | first variant copy 1 | 25.0% | 1.6% | 10.4% |
This experiment has a required sample size of 95,144 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
The treatment produced no significant impact—increase or decrease—in conversion. Further data analysis showed that 73% of people who saw the treatment also opted into the Hoover Daily Report, which provided additional cultivation and engagement.
Ultimately, though there wasn’t a lift, this test was a win—conversion rate was not affected, and most users chose to receive additional content from Hoover Institution.