The impact a Yes instead of a Yes/No opt-in option on a landing page Experiment ID: #9462

Care Net

Experiment Summary

Ended On: 9/27/2018

We had just finished testing a set of checkboxes offering leads the opportunity to opt-in to a workflow designed to nurture them to becoming monthly donors, Caring Partners. Our test found that these checkboxes reduced conversion by 15%, but 11% of signups opted into the workflow.

We then wanted to test if we could get more opt-ins by making it a single checkbox for opting in. The thought was that introducing the negative response as an option could cause more people to respond negatively.

Research Question

Will giving people a checkbox for opting in, but not one for opting out, increase the number of leads opting in?

Design

C: Two checkboxes
T1: single checkbox

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Two checkboxes 31.1%
T1: single checkbox 22.6% -27.2% 100.0%

This experiment has a required sample size of 214 in order to be valid. Since the experiment had a total sample size of 4,815, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 27.2% decrease in conversion rate
× 0% increase in average gift

Key Learnings

Giving visitors only one checkbox, instead of a yes or no had a significant decrease in the number of Caring Partner Opt-Ins. Our hypothesis is that the two checkboxes allowed visitors to give more extensive feedback and was not as easily ignored as the single checkbox. It was also interesting to note that neither variation had a significant difference in the number of emails acquired, the impact was solely on the number of leads opting in.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.