How adding an extra step in an acquisition funnel affected donor conversion
Bill of Rights Institute
Experiment Summary
Timeframe: 07/10/2020 - 07/30/2020
BRI was running an offer for a quiz about Natural Rights. They also were running an offer for an eBook about Natural Rights. So we combined the two into a treatment; when someone took the quiz they were then also offered the eBook. This did mean that we asked them to go through one more step and would potentially lose new names. However, we wanted to see if increasing the value of the offer through an eBook would increase donor conversion.
Research Question
Does adding an eBook offer to a quiz convert donors better, even with fewer people completing the funnel because it has more steps?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Quiz to Donation Page | 0.02% | ||
| T1: | Quiz to eBook to Donation Page | 0.07% | 184.1% | 99.8% |
This experiment has a required sample size of 17,977 in order to be valid. Since the experiment had a total sample size of 83,849, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 184.1% increase in conversion rate
× 0% increase in average gift
Key Learnings
The donor conversion rate was 184% higher with a 99.7% Level of Confidence. The average gift was higher by $4, at an $18.70 average gift for the quiz alone and $22.79 for the quiz plus eBook. The revenue also saw a 246% increase at a 99.7% LoC.
The conversion rate on the landing page for the eBook was lower than the conversion on the quiz alone. The quiz funnel had a 3% email conversion rate and the quiz to eBook treatment had a 2.7% email conversion rate, resulting in approximately 131 fewer emails acquired.
The ROI was greatly improved — .46 ROI over .11 ROI, a 318% increase. In fact, during this span of time, the ROI for the quiz to eBook funnel was higher than simply the eBook funnel, which until then had the highest results.
However, because the ideal number of total conversions was not achieved, we do recommend further testing in this area. We intend to run an iteration of this test is to find out which converts donors better—the longer funnel with more engagement and value or the shorter funnel with the just eBook and less friction.