How does paginating a multi-section survey affect email acquisition rates?
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Timeframe: 06/27/2022 - 07/10/2022
In an effort to improve email acquisition rates on a multi-section survey offer, we wanted to test whether or not breaking a long 19-question survey into more manageable, less-intimidating chunks would decrease the perceived “cost” of completing the survey, in terms of time, and decrease perceived friction enough to improve sign-up rates.
We ran this experiment for 28 days, splitting traffic 50/50 between the control and the treatment.
Research Question
We believe that paginating a multi-section survey for potential donors will achieve higher email acquisition rates.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | 16.7% | $0.00 | ||
| T1: | Paginated Version | 41.7% | 150.0% | 86.5% | $0.00 |
This experiment has a required sample size of 24 in order to be valid. Since the experiment had a total sample size of 30, and the level of confidence is not above 95% the experiment results are not valid.
Key Learnings
After 28 days, we observed a 31.5% lift in email acquisition rates, with a 97% level of confidence, validating our hypothesis that by reducing the perceived cost of completing the survey, in terms of time, visitors would experience less friction and would be more motivated to complete a multi-section, 19-question survey.
The results of this experiment suggest that presenting all 19 survey questions on one long page may have made completing the survey a long and laborious task, increasing the perceived cost (time) of taking the survey to a point where it outweighed the perceived value in the minds of prospects.
By breaking the survey into more manageable chunks, with each section on its own page, the perceived cost of completing the survey was reduced in the minds of prospects while the persuasion principle of ‘behavioral consistency’ likely helped to motivate prospects to complete the survey.
For mobile users, the experiment produced a 100.3% lift in acquisition rates, with a 100% level of confidence, likely due to the decreased scroll and potential “gamification” presented by the paginated version.
Based on these results, we recommend that organizations with longer survey offers consider paginating such surveys to reduce the perceived cost and leverage the behavioral principle of ‘behavioral consistency’ to lift acquisition rates.