How pre-checking an email opt-in box affects email opt-ins Experiment ID: #20421

CaringBridge

CaringBridge offers free personal, protected websites for people to easily share updates and receive support and encouragement from their community during a health journey. Every 7 minutes, a CaringBridge website is created for someone experiencing a health event.

Experiment Summary

Ended On: 3/23/2020

As part of their new user registration process, CaringBridge had always given people the option to opt-in to marketing emails, new content, stories, and helpful articles as they created their account. However, that check box had always been unchecked.

Research Question

How will pre-checking an email opt-in box affect email opt-in rate?

Design

C: Control
T1: Treatment #1

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Control 8.1%
T1: Treatment #1 18.3% 126.7% 100.0%

This experiment has a required sample size of 84 in order to be valid. Since the experiment had a total sample size of 90,341, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 126.7% increase in conversion rate
× 0% increase in average gift

Key Learnings

The treatment, with the pre-checked box, resulted in a 126.7% increase in email address acquisition rate. This showed that the small act of checking a box was a tremendous point of friction to users, who might otherwise be amenable to opting in. This showed that the cost of checking the box was too high—which had nothing to do with the value (or perceived value) of the subscription. Once the cost was taken care of, the user was much more likely to accept the value.


Experiment Documented by...

Jeff Giddens

Jeff is the President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.