How adding an extra step in an acquisition funnel affected donor conversion Experiment ID: #21603

Bill of Rights Institute

Experiment Summary

Timeframe: 7/10/2020 - 7/30/2020

BRI was running an offer for a quiz about Natural Rights. They also were running an offer for an eBook about Natural Rights. So we combined the two into a treatment; when someone took the quiz they were then also offered the eBook. This did mean that we asked them to go through one more step and would potentially lose new names. However, we wanted to see if increasing the value of the offer through an eBook would increase donor conversion.

Research Question

Does adding an eBook offer to a quiz convert donors better, even with fewer people completing the funnel because it has more steps?

Design

C: Quiz to Donation Page
T1: Quiz to eBook to Donation Page

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Quiz to Donation Page 0.02%
T1: Quiz to eBook to Donation Page 0.07% 184.1% 99.8%

This experiment has a required sample size of 17,977 in order to be valid. Since the experiment had a total sample size of 83,849, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 184.1% increase in conversion rate
× 0% increase in average gift

Key Learnings

The  donor conversion rate was 184% higher with a 99.7% Level of Confidence. The average gift was higher by $4, at an $18.70 average gift for the quiz alone and $22.79 for the quiz plus eBook. The revenue also saw a 246% increase at a 99.7% LoC.

The conversion rate on the landing page for the eBook was lower than the conversion on the quiz alone. The quiz funnel had a 3% email conversion rate and the quiz to eBook treatment had a 2.7% email conversion rate, resulting in approximately 131 fewer emails acquired.

The ROI was greatly improved — .46 ROI over .11 ROI, a 318% increase. In fact, during this span of time, the ROI for the quiz to eBook funnel was higher than simply the eBook funnel, which until then had the highest results.

However, because the ideal number of total conversions was not achieved, we do recommend further testing in this area. We intend to run an iteration of this test is to find out which converts donors better—the longer funnel with more engagement and value or the shorter funnel with the just eBook and less friction.


Experiment Documented by...

Gina Gallutia

Gina is . If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.