How the total number of questions in a quiz affects email acquisition Experiment ID: #8057

Illinois Policy Institute

Experiment Summary

Timeframe: 11/13/2017 - 12/8/2017

This year, Illinois Policy Institute has exposed a serious problem in the state of Illinois. It’s been more than 15 years since the Illinois General Assembly has passed a balanced budget, and lawmakers have turned to tax hikes in order to ease the deficit. However, this is not a long-term solution and it’s caused illinoisans to pay some of the highest taxes in the nation. Illinois Policy developed a plan to balance the budget and ease taxes, all at once. In order to get the word out, they developed a 9-question quiz followed by a tax petition that would build awareness among Illinoisans and encourage them to take action to advance this plan. Before launching, Illinois Policy Institute wondered: Will a shorter, 6-question quiz increase petition signatures?

They created a treatment version and launched an A/B test to find out.

Research Question

Will a shorter quiz increase email acquisition rate?

Design

C: Control
T1: Treatment 1

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Control 31.7%
T1: Treatment 1 32.8% 3.7% 84.4%

This experiment has a required sample size of 12,139 in order to be valid. Since the experiment had a total sample size of 12,700, and the level of confidence is not above 95% the experiment results are not valid.

Key Learnings

Illinois Policy Institute found that the shorter quiz did not produce a valid increase or decrease in email acquisition. While these results aren’t statistically significant, there is an important takeaway. We’ve seen a correlation in other testing that shows the longer time commitment an individual makes in consuming an offer, the higher their instant and long-term donor conversion rate is. Since their is no significant difference between email acquisition rate in this test, it’s possible that we can increase donor conversion by continuing to offer the longer quiz (i.e. the one that requires more of a time commitment). The next step for this test sequence is to test an even longer version of the quiz to see if we can impact donor conversion without hurting email acquisition.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.