How the way the gift frequency section on the the primary donation page is presented impacts conversion during year-end
Focus on the Family
Focus on the Family is a global Christian ministry dedicated to helping families thrive. We provide help and resources for couples to build healthy marriages that reflect God's design, and for parents to raise their children according to morals and values grounded in biblical principles.
Experiment Summary
Timeframe: 12/23/2020 - 01/08/2021
Focus on the Family recently changed their donation platform. During that transition, the way the gift frequency section was presented looked and functioned differently than before. They heard from prospective donors that it was slightly confusing for them as they made a gift. To test into the best approach for that section on the page, we developed two different variations in hopes that we could increase clarity to the giving process. The first variation presented the two different gift frequency options as checked boxes – requiring the prospective donor to have to select the gift they wanted to make through the checked boxes. The second variation presented the options as buttons. Again, requiring the prospective donor to have to select the gift they wanted by selecting the specific button. This experiment was run during the last few weeks of their calendar year-end campaign.
Research Question
We believe that changing the way the gift frequency section was presented and functioned for prospective donors during year-end will achieve an increase in donor conversion.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 32.4% | ||
| T1: | Checkbox | 33.9% | 4.6% | 94.1% |
| T2: | Buttons | 32.8% | 1.2% | 36.6% |
This experiment has a required sample size of 11,450 in order to be valid. Since the experiment had a total sample size of 21,307, and the level of confidence is not above 95% the experiment results are not valid.
Key Learnings
While the experiment didn’t validate, we did see a directional lift in donor conversion from the checked boxes variation, and no difference in conversion from the button variation.
One of the big questions around this experiment was the impact the variations had on recurring giving . We hypothesized that the control variation may see an increase in recurring because of the lack of clarity it presented to donors about how they could make a one-time gift. In analyzing that data, we found that the control did see a significant increase in recurring donor conversion. In fact, the checked box variation decreased recurring transactions by 41% and the button variation decreased recurring transactions by 33% (88% valid). That said, the organization is seeing an increase in recurring gift cancellations right now. We hypothesize that this is because of the confusion that the control may have been causing in the kind of gift a donor was making.
Additionally, this experiment was run during a time of year when motivation of donors is at its highest. Because of this and the impact we saw around recurring gifts, we are retesting this approach this month so that we can have a larger data sample and more neutral motivation.