How longer value proposition on an acquisition donation page increases revenue
Bill of Rights Institute
Experiment Summary
Timeframe: 03/11/2021 - 05/09/2021
The Bill of Rights Institute had been running an offer for an eBook on Equal Rights for months. And the average gift was not as high as we believed it could be. So we decided to test including more copy on the donation page around the need for their gift and the impact it would make. The hypothesis was that this would lead them to give more.
This test ran for two months, longer than typically suggested, because we continued to see fluctuations in the results and wanted to give it a long time horizon.
Research Question
Expanding value proposition on an acquisition donation page will increase revenue by increasing the perceived value of their gift.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | 4.3% | $0.00 | ||
| T1: | Extended Value Prop | 4.2% | -1.4% | 3.5% | $0.00 |
This experiment has a required sample size of 822,933 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
This test did not result in valid findings. When looking at the different demographics. We found that it increased revenue with new visitors by 45% at a 87% LOC, and decreased the revenue from returning visitors by 24% at a LOC of 77%. The findings from these two segments of users negated the other when you view this test through all traffic.
Our hypothesis is that this test showed a directional lift in revenue for new visitors because it told them more about an organization that they are new to, and it taught them more about the value of the eBook they just received. But overall, this test showed that this copy alone was not enough to increase revenue significantly for a sustainable amount of time.
Despite the extra time, these findings were still not strong enough to roll out this test. We will be testing other avenues for increasing revenue.