How reducing friction by shortening survey length impacted email acquisition
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Timeframe: 02/06/2023 - 03/17/2023
When reviewing results of a survey offer funnel, we observed a solid click-through rate, a low cost per click, and a fair instant conversion rate indicating that this is an offer and an issue people care to engage with. However, we identified lower-than-average email acquisition rates as an opportunity to improve overall results. After reviewing this campaign, we believe that the length of the survey was introducing significant friction and causing visitors to abandon. In all, the survey contains 20 questions. In a previous experiment, we were able to produce a sizable lift in survey completions by paginating the survey, rather than presenting all 20 questions on one page. However, over time, email acquisition was still struggling for this offer and we believe that its length was causing the cognitive momentum in landing page visitors to fizzle out. To test this theory, we built a treatment that truncated the survey to ~5 questions all presented on one page to reduce friction/cost of completing survey in effort to improve email acquisition. We split traffic 50/50 between the two versions and ran the experiment for one month.
Research Question
We believe that reducing the length of a survey will increase email acquisition rates by reducing friction and capitalizing on the cognitive momentum created by the ad that sends traffic to the landing page.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | 38.2% | $0.00 | ||
| T1: | 5 Question Version | 38.6% | 0.9% | 12.4% | $0.00 |
This experiment has a required sample size of 167,516 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
After running the experiment for one month we saw a 0.86% increase in conversion rate with 12.4% level of confidence. However, we learned that not all friction is bad friction because we saw a significant decrease in Donations and Revenue, with a 100% and 98% level of confidence, respectively. This suggests that reducing the length of the survey may have a negative impact on the willingness of visitors to donate or contribute financially. When reviewing downstream effects on donations, the treatment produced a 90% decrease in gifts with 99% level of confidence. We believe this happened because the more time that was invested in the survey, the more willing the visitor was to convert on the instant donation page. This means the control version with a lengthier survey was actually producing more qualified traffic to the donation page. Because the results were overall flat at the landing page level, we will need to do additional testing to improve email acquisition rate while still maintaining the quality of the visitor at the donation page level. In future testing, we could add additional elements to the process to let visitors know how many steps there are to complete the survey and display their progress along the way so they have more clarity around how long it will take them to finish the survey and submit their responses. It is important to note that we saw a significant increase in email acquisition rates for Housefile Email visitors, suggesting that the treatment may have a positive impact on visitors who are already familiar with our organization. Future experimentation should focus on finding a balance between reducing friction and maintaining the willingness of visitors to donate or contribute financially. We can also consider targeting Housefile Email visitors with a truncated survey to capitalize on their familiarity with our organization and potentially increase email acquisition rates.