Care Net

How adding clarity around the length of a survey impacted clicks from non-donors

Experiment ID: #160360

Care Net

Experiment Summary

Timeframe: 09/06/2023 - 09/08/2023

When sending a survey to a non-donor audience, we wondered if including the number of questions in a survey within an email would enhance its appeal and clarity for recipients. By stating the number of questions upfront, recipients can gauge the time commitment required, making the survey seem more manageable and less daunting. This transparency builds trust and sets clear expectations, making the value proposition of participating in the survey more compelling. It assures recipients that their time is valued and that the survey won’t be overly lengthy or burdensome.

Research Question

We believe that clearly communicating the length of a survey for non-donors will achieve an increase in survey participation because of the enhanced appeal and clarity of the time and effort involved..

Design

C: Version A

T1: Version B

Results

	Treatment Name	Click Rate	Relative Difference	Confidence
C:	Version A	0.43%
T1:	Version B	0.57%	33.1%	99.9%

This experiment has a required sample size of 19,175 in order to be valid. Since the experiment had a total sample size of 118,517, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

33.1% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The key learnings from this experiment are:

1. Clearly communicating the length of a survey upfront enhances its appeal and clarity for recipients.

2. Providing the number of questions in the survey helps recipients gauge the time commitment required, making it seem more manageable and less daunting.

3. Transparency builds trust and sets clear expectations, making the value proposition of participating in the survey more compelling.

4. Clearly communicating the length of the survey resulted in a significant increase in both opens and clicks.

5. The treatment outperformed the control group with a 33.1% increase in Clicks for All Traffic with 99% level of confidence.

Based on these learnings, future experimentation should continue to focus on clear communication and setting expectations with recipients. Testing variations in the way the length of surveys is communicated, such as using different formats or emphasizing the time commitment in different ways, can provide further insights into what resonates most effectively with the target audience.

Additionally, the significant increases in Clicks for All Traffic indicate that clearly communicating the length of a survey motivates more people to click on the survey link if the perceived effort required seems low. Exploring other elements of the email or survey content that could enhance click-through rates can be valuable for future experimentation.

Overall, this experiment highlights the importance of transparency and setting clear expectations when communicating with non-donor audiences. By incorporating these learnings into future experimentation, Care Net can continue to optimize engagement and ultimately increase survey participation.

Experiment Documented by Rebekah Josefy

Rebekah Josefy is an Optimization Director at NextAfter.

Question about experiment #160360

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.