How qualifying questions affects conversion rate - NextAfter
Jews for Jesus

How qualifying questions affects conversion rate

Experiment ID: #2086

Jews for Jesus

Jews for Jesus exists to make the messiahship of Jesus an unavoidable issue to the Jewish people worldwide, utilizing creative evangelistic methods to reach the Jewish community and spread the good news of Jesus Christ.

Experiment Summary

Timeframe: 09/03/2015 - 09/17/2015

Jews for Jesus produces a variety of different publications that are tailored specifically for the different segments of their email file. Since their mission is to reach the unbelieving Jewish population, they want to make sure they do not inadvertently solicit money from nonbelievers. They also want to make sure the publications are relevant to the reader.

The email acquisition section of their homepage was traditionally just an email address and we made the assumption that those people were believers. Knowing that this isn’t always the case, we decided to add some qualifying questions to the acquisition form so that we could correctly segment the audience. However, we wanted to make sure this didn’t adversely impact the number of emails acquired on the page so we create an experiment to measure the results.

Research Question

How will the qualifying questions impact the email acquisition rate of the homepage?

Design

C: No Questions
T1: Religious Questions

Results

  Treatment Name Conv. Rate Relative Difference Confidence
C: No Questions 0.77%
T1: Religious Questions 1.5% 89.3% 98.6%

This experiment has a required sample size of 1,806 in order to be valid. Since the experiment had a total sample size of 5,614, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 89.3% increase in conversion rate
× 0% increase in average gift

Key Learnings

The results of the experiment were surprising; by asking the qualifying questions: “Do you believe in Jesus?” and “Are you Jewish?” we were able to increase the number of email addresses acquired by 89%.

Often times, qualifying questions like this can add unnecessary friction to the process. However, in this situation, our hypothesis is that people interpreted these questions as the organization wanting to get to know them better.  As a result, they saw them as adding value to the publications they would receive which then increased their likelihood of signing up.


Experiment Documented by NextAfter

Question about experiment #2086

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.