How qualifying questions affects conversion rate
Jews for Jesus
Jews for Jesus exists to make the messiahship of Jesus an unavoidable issue to the Jewish people worldwide, utilizing creative evangelistic methods to reach the Jewish community and spread the good news of Jesus Christ.
Experiment Summary
Timeframe: 09/03/2015 - 09/17/2015
Jews for Jesus produces a variety of different publications that are tailored specifically for the different segments of their email file. Since their mission is to reach the unbelieving Jewish population, they want to make sure they do not inadvertently solicit money from nonbelievers. They also want to make sure the publications are relevant to the reader.
The email acquisition section of their homepage was traditionally just an email address and we made the assumption that those people were believers. Knowing that this isn’t always the case, we decided to add some qualifying questions to the acquisition form so that we could correctly segment the audience. However, we wanted to make sure this didn’t adversely impact the number of emails acquired on the page so we create an experiment to measure the results.
Research Question
How will the qualifying questions impact the email acquisition rate of the homepage?
Design
Results
Treatment Name | Conv. Rate | Relative Difference | Confidence | |
---|---|---|---|---|
C: | No Questions | 0.77% | ||
T1: | Religious Questions | 1.5% | 89.3% | 98.6% |
This experiment has a required sample size of 1,806 in order to be valid. Since the experiment had a total sample size of 5,614, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 89.3% increase in conversion rate
× 0% increase in average gift
Key Learnings
The results of the experiment were surprising; by asking the qualifying questions: “Do you believe in Jesus?” and “Are you Jewish?” we were able to increase the number of email addresses acquired by 89%.
Often times, qualifying questions like this can add unnecessary friction to the process. However, in this situation, our hypothesis is that people interpreted these questions as the organization wanting to get to know them better. As a result, they saw them as adding value to the publications they would receive which then increased their likelihood of signing up.
Question about experiment #11629
If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.