How adding an additional cultivation opportunity affects email acquisition Experiment ID: #10736

Hoover Institution

Experiment Summary

Timeframe: 5/7/2018 - 5/11/2018

Hoover Institution was running their Make America Exceptional Again offer, and wanted to know the effects of offering a checkbox to sign up for the HDR newsletter by the bottom of the form. All copy remained the same. 

Research Question

Does adding a checkbox to receive additional content at the end of a form affect conversion rate? 


C: No Newsletter Opt-In
T1: Newsletter Opt-In


Treatment Name Conv. Rate Relative Difference Confidence
C: No Newsletter Opt-In 24.5%
T1: Newsletter Opt-In 25.0% 2.0% 13.2%

This experiment has a required sample size of 57,976 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

This experiment did not experience a statistically valid lift. However, this experiment not statistically validating is just as important as if we had seen a lift. This means we can increase the subscribers to the newsletter without adversely impacting acquisition.

The goal of these email acquisition campaigns is to create a file of prospective donors that can also be converted downstream. The best way to do that is through ongoing cultivation.

By getting a new subscriber to opt-in to the ongoing newsletter cultivations, we are putting them into a communication flow that will increase the likelihood of downstream conversion. Since they are having to physically opt-in to the series, it will also increase their perceived value of the communications they will receive.

Experiment Documented by...

Allison Jones

Allison is an Optimization Associate at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.