The Heritage Foundation

How does paginating a multi-section survey affect email acquisition rates?

Experiment ID: #98942

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Timeframe: 06/27/2022 - 07/10/2022

In an effort to improve email acquisition rates on a multi-section survey offer, we wanted to test whether or not breaking a long 19-question survey into more manageable, less-intimidating chunks would decrease the perceived “cost” of completing the survey, in terms of time, and decrease perceived friction enough to improve sign-up rates.

We ran this experiment for 28 days, splitting traffic 50/50 between the control and the treatment.

Research Question

We believe that paginating a multi-section survey for potential donors will achieve higher email acquisition rates.

Design

C: Control
T1: Paginated Version

Results

  Treatment Name Conv. Rate Relative Difference Confidence Average Gift
C: Control 16.7% $0.00
T1: Paginated Version 41.7% 150.0% 86.5% $0.00

This experiment has a required sample size of 24 in order to be valid. Since the experiment had a total sample size of 30, and the level of confidence is not above 95% the experiment results are not valid.

Key Learnings

After 28 days, we observed a 31.5% lift in email acquisition rates, with a 97% level of confidence, validating our hypothesis that by reducing the perceived cost of completing the survey, in terms of time, visitors would experience less friction and would be more motivated to complete a multi-section, 19-question survey.

The results of this experiment suggest that presenting all 19 survey questions on one long page may have made completing the survey a long and laborious task, increasing the perceived cost (time) of taking the survey to a point where it outweighed the perceived value in the minds of prospects.

By breaking the survey into more manageable chunks, with each section on its own page, the perceived cost of completing the survey was reduced in the minds of prospects while the persuasion principle of ‘behavioral consistency’ likely helped to motivate prospects to complete the survey.

For mobile users, the experiment produced a 100.3% lift in acquisition rates, with a 100% level of confidence, likely due to the decreased scroll and potential “gamification” presented by the paginated version.

Based on these results, we recommend that organizations with longer survey offers consider paginating such surveys to reduce the perceived cost and leverage the behavioral principle of ‘behavioral consistency’ to lift acquisition rates.


Experiment Documented by NextAfter

Question about experiment #98942

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.