How a Wikipedia-style ask performs against a banner ad and a slide-in appeal on article pages during a high-urgency campaign
Focus on the Family
Focus on the Family is a global Christian ministry dedicated to helping families thrive. We provide help and resources for couples to build healthy marriages that reflect God's design, and for parents to raise their children according to morals and values grounded in biblical principles.
Experiment Summary
Timeframe: 03/02/2023 - 03/16/2023
During past high-urgency campaigns, an organization we work with has traditionally run a banner appeal on article pages in an attempt to increase giving from traffic who visit these articles.
Because this organization earns a high volume of traffic to the article pages, we wanted to test the banner appeal against two treatments that we have seen success with in the past to test how they would perform against one another in the context of a high-urgency campaign specifically soliciting recurring donations.
We felt that the banner appeal, while visually attractive and prevalent on the page, lacked a value proposition that was compelling enough to inspire people to make the commitment to give a monthly recurring gift. Both treatments employed nearly identical language both speaking to the problem/solution being conveyed by the appeal and the urgency created by the fact that only 6% of all donors give on a monthly.
The two tactics we wanted to test were a) a Wikipedia-inspired top-of-page appeal that converts to a slide-up appeal when the user scrolls and b) a slide-in appeal that appears from the right side of the screen at about 30% page depth.
For this experiment, we split traffic between the banner appeal (control), the Wikipedia-inspired appeal (treatment A), and the slide-in appeal (treatment B), splitting traffic roughly evenly between each. It was our hypothesis that given its visibility and the appeal it communicates, the Wikipedia inspired ask would perform best.
Research Question
We believe that a Wikipedia-style banner on article pages will increase recurring donations during a high-urgency campaign because it presents a more compelling appeal in a prevalent location, leveraging urgency and reciprocity to inspire people to give.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | 0.02% | |||
| T1: | Wikipedia | 0.06% | 156.9% | 99.8% | |
| T2: | Slide-in | 0.01% | -35.3% | 69.6% |
This experiment has a required sample size of 20,481 in order to be valid. Since the experiment had a total sample size of 195,773, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 156.9% increase in conversion rate
× 0% increase in average gift
Key Learnings
After running this experiment for two weeks, we were able to validate a 156.9% increase in recurring donations produced by the Wikipedia-inspired banner appeal (99.8% level of confidence). In addition to this encouraging lift in recurring donations, we also observed:
- 582.2% in desktop recurring donations (34 vs. 5), 99.9% LoC
- 75.5% in mobile recurring donations (14 vs. 8), 80% LoC
- 28.1% in total donations, 89.5% LoC
These results suggest that the Wikipedia-inspired appeal is most effective in engaging visitors and compelling them to donate on a recurring basis.
We believe that the more compelling and thorough language used in the Wikipedia-inspired appeal—detailing the urgency of the problem, conveying the solution the donor can help achieve, and leveraging the persuasion principle of consistency by conveying that only 6% of donors give monthly—led to the increase in recurring gifts as compared to the control (banner).
And we believe that the more prominent location of the Wikipedia-inspired appeal led to the increase in recurring gifts as compared to treatment B (slide-in).
This experiment further validates the effectiveness of the Wikipedia-style banner tactic. Now, future experimentation can broaden this learning by investigating which persuasion principles work best in this context to improve on these results.