Luther Seminary

How using No-primed questions in a survey impacted engagement

Experiment ID: #195809

Luther Seminary

Experiment Summary

Timeframe: 11/18/2024 - 11/27/2024

During high-urgency campaigns such as calendar year-end, we often use a survey as a tool to help “prime” Luther Seminary’s audience with knowledge about the problems that Luther Seminary is working to solve through their fundraising campaign. Typically, these survey questions are “yes-primed”, which means that they’re designed so that a Luther Seminary supporter would respond “Yes” to each question.

For Luther Seminary’s calendar year-end campaign, we wanted to test this “yes-primed” framing against a survey that was “no-primed”. Our goal was to understand how this “Yes” or “No” priming impacts engagement and downstream donor behavior.

Both survey funnels were identical, with the only change being the phrasing of the 5 survey questions. The Control survey used “Yes”-primed questions, which are designed to elicit affirmative responses from respondents. The Treatment survey, on the other hand, used “No”-primed questions, encouraging respondents to think critically about the topics and more likely to answer negatively.

This experiment aimed to explore the psychological principle of reactance theory, which suggests that when people feel their freedom of choice is being restricted or guided (such as through overly positive framing), they may react by resisting or disengaging. Conversely, by using “No”-primed questions, we hypothesized that respondents would feel a greater sense of autonomy and control over their answers. The idea is that when people are encouraged to think critically and reject easy affirmations, they may feel more invested in their responses, fostering a stronger connection with the survey.

Additionally, answering “No” to these questions may create a cognitive dissonance effect, where respondents feel the need to resolve the negative stance with a positive action, such as donating after completing the survey. This subconscious desire for resolution may lead to higher engagement during the survey and increased donor conversions later in the funnel, especially when presented with an opportunity to act on their beliefs.

We split traffic to the survey pages evenly, and measured the # of survey submissions (emails acquired), as well as downstream donations on the post-survey donation page.

Research Question

We believe that using “No-primed” questions for a survey offer will achieve an increase in survey completions (email submissions) because respondents will feel a greater sense of autonomy and control over their answers, which may lead to a subconscious desire to resolve with a “Yes” when asked to contribute later.

Design

C: Control - "Yes-Primed"

T1: Treatment - "No-Primed"

Results

	Treatment Name	Conv. Rate	Relative Difference	Confidence
C:	Control - "Yes-Primed"	42.2%
T1:	Treatment - "No-Primed"	45.5%	7.8%	79.1%

This experiment has a required sample size of 1,746 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

The results of this test can be broken down into 3 key learnings: #1 - Increase in Engagement with “No”-Primed Questions The Treatment survey, which used "No"-primed questions, resulted in a 7.8% increase in form submissions compared to the Control survey. While this increase did not meet the 95% statistical level of confidence (79.1% LOC), it still suggests a promising trend toward higher engagement. This aligns with our hypothesis, which was grounded in reactance theory. By encouraging respondents to critically evaluate the questions and answer negatively, we likely fostered a greater sense of autonomy. Respondents who felt they were not being led to affirmations may have felt more invested in their responses, leading to increased engagement. #2 - Potential Impact of “No” Priming on Conversion Behavior Despite the increase in form submissions, the Conversion Rate for the Treatment group (2.41%) was slightly lower than the Control group (3.36%). This discrepancy suggests that while “No”-primed questions led to more people completing the survey, there might be a short-term resistance to following through with donations immediately after completing the survey. However, the cognitive dissonance effect mentioned in our hypothesis may still be at play—respondents who answered "No" to the questions may feel a subconscious need to resolve the negativity, and this could influence them to donate in the future as they reconcile their answers with the desire to support Luther Seminary. #3 - Higher Average Gift in the Treatment Group The Treatment group had a higher average gift of $32, compared to the Control group’s $23. This finding suggests that while the number of donors was slightly lower in the Treatment group, those who did choose to give were more likely to contribute larger amounts. This is potentially indicative of commitment consistency. Respondents who were engaged through the "No"-primed survey, and who felt more autonomous in their responses, may have been more motivated to make a larger, more deliberate contribution to Luther Seminary's cause. By thinking critically about their answers, they may have been more intentional in their giving. The results of this experiment point toward the potential effectiveness of “No”-primed surveys for increasing engagement and stimulating larger gifts, even if immediate conversions didn’t significantly increase. Respondents who answered negatively may have felt more empowered and invested in their decisions, which could lead to higher-value gifts in the long term. Future tests could explore the extended impact of these psychological principles, with a focus on donor behavior over time and refining the optimal balance between engagement and conversion.

Experiment Documented by NextAfter

Question about experiment #195809

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.