American Cornerstone Institute

How adding privacy language below a survey sign up form impacted sign up rate

Experiment ID: #146608

American Cornerstone Institute

Experiment Summary

Timeframe: 04/25/2023 - 05/28/2023

ACI’s Education Survey is producing a 43.3% sign up rate over the last month. While this metric is performing above-benchmark, we believe there is room for improvement for this offer to match the sign up rates of other offers such as the Cancel Culture Pledge (59.5% sign up rate).

In reviewing the current landing page, we wonder if emphasizing the privacy language below the form will build trust with our audience and lead a higher percentage of our audience to submit their survey. We’ve seen this tactic work across other similar organizations.

People can feel vulnerable giving away their information, so adding this language could provide them more comfort and trust and it may be the added punch that this page needs to improve sign up rate.

Research Question

We believe that including a more formal privacy statement at the bottom of the form will increase the education survey sign up rate because it will establish more credibility and trust in ACI and the survey itself.

Design

C: Control

T1: Variant 1

Results

	Treatment Name	Conv. Rate	Relative Difference	Confidence	Average Gift
C:	Control	42.5%			$0.00
T1:	Variant 1	40.7%	-4.3%	99.5%	$0.00

This experiment has a required sample size of 5,518 in order to be valid. Since the experiment had a total sample size of 22,274, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

0% increase in traffic
× 4.3% decrease in conversion rate
× 0% increase in average gift

Key Learnings

The key learning from this experiment is that including a more formal privacy statement at the bottom of the form did not increase the Education Survey sign up rate for ACI. The treatment saw a 4.3% decrease in Email Acquisition for All Traffic with a 99.5% statistical level of confidence, indicating that the control version performed better.

An important metric to review regarding data privacy is new visitors, as they may not have strong familiarity with the organization they'd be giving their information to. In this case, new visitors produced a valid 4.7% decrease in sign up rate, showing that they were actually less likely to submit their information when viewing the data privacy language.

Interestingly, donation revenue increased by 23.7% with an 84.1% level of confidence. Although it did not validate, the results are encouraging and show that those who did build trust through the landing page privacy language did give more once they reached the donation page.

The downstream metrics tell us that we're onto something with this approach. So as a next test, we will test shorter, more direct privacy language and place it below the submit button as a way to maintain the user momentum. We hypothesize that because the privacy language was placed between the form and the submit button for this experiment, this may have introduced friction, making users think twice about submitting their information. The language should compliment the sign up process instead of impede it.

Experiment Documented by NextAfter

Question about experiment #146608

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.