The impact a Yes instead of a Yes/No opt-in option on a landing page
Care Net
Experiment Summary
Ended On: 09/27/2018
We had just finished testing a set of checkboxes offering leads the opportunity to opt-in to a workflow designed to nurture them to becoming monthly donors, Caring Partners. Our test found that these checkboxes reduced conversion by 15%, but 11% of signups opted into the workflow.
We then wanted to test if we could get more opt-ins by making it a single checkbox for opting in. The thought was that introducing the negative response as an option could cause more people to respond negatively.
Research Question
Will giving people a checkbox for opting in, but not one for opting out, increase the number of leads opting in?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Two checkboxes | 31.1% | ||
| T1: | single checkbox | 22.6% | -27.2% | 100.0% |
This experiment has a required sample size of 214 in order to be valid. Since the experiment had a total sample size of 4,815, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 27.2% decrease in conversion rate
× 0% increase in average gift
Key Learnings
Giving visitors only one checkbox, instead of a yes or no had a significant decrease in the number of Caring Partner Opt-Ins. Our hypothesis is that the two checkboxes allowed visitors to give more extensive feedback and was not as easily ignored as the single checkbox. It was also interesting to note that neither variation had a significant difference in the number of emails acquired, the impact was solely on the number of leads opting in.