How the total number of questions in a quiz affects email acquisition
Illinois Policy Institute
Experiment Summary
Timeframe: 11/13/2017 - 12/08/2017
This year, Illinois Policy Institute has exposed a serious problem in the state of Illinois. It’s been more than 15 years since the Illinois General Assembly has passed a balanced budget, and lawmakers have turned to tax hikes in order to ease the deficit. However, this is not a long-term solution and it’s caused illinoisans to pay some of the highest taxes in the nation. Illinois Policy developed a plan to balance the budget and ease taxes, all at once. In order to get the word out, they developed a 9-question quiz followed by a tax petition that would build awareness among Illinoisans and encourage them to take action to advance this plan. Before launching, Illinois Policy Institute wondered: Will a shorter, 6-question quiz increase petition signatures?
They created a treatment version and launched an A/B test to find out.
Research Question
Will a shorter quiz increase email acquisition rate?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 31.7% | ||
| T1: | Treatment 1 | 32.8% | 3.7% | 84.4% |
This experiment has a required sample size of 12,139 in order to be valid. Since the experiment had a total sample size of 12,700, and the level of confidence is not above 95% the experiment results are not valid.
Key Learnings
Illinois Policy Institute found that the shorter quiz did not produce a valid increase or decrease in email acquisition. While these results aren’t statistically significant, there is an important takeaway. We’ve seen a correlation in other testing that shows the longer time commitment an individual makes in consuming an offer, the higher their instant and long-term donor conversion rate is. Since their is no significant difference between email acquisition rate in this test, it’s possible that we can increase donor conversion by continuing to offer the longer quiz (i.e. the one that requires more of a time commitment). The next step for this test sequence is to test an even longer version of the quiz to see if we can impact donor conversion without hurting email acquisition.