How adding impact language to a direct-ask pop up banner during a high urgency campaign affects revenue – marriages saved V1
Focus on the Family
Focus on the Family is a global Christian ministry dedicated to helping families thrive. We provide help and resources for couples to build healthy marriages that reflect God's design, and for parents to raise their children according to morals and values grounded in biblical principles.
Experiment Summary
Timeframe: 11/23/2022 - 12/02/2022
Focus on the Family (FOTF) is a global Christian ministry dedicated to helping families thrive. They provide help and resources for couples to build healthy marriages that reflect God’s design and for parents to raise their children according to morals and values grounded in biblical principles. Every year during the calendar year-end high-urgency campaign, FOTF places a top-of-page pop-up banner that contains a direct ask to support their mission. This banner (known internally as the “wiki-banner”) has been tested into over many years, with experiments to improve donation metrics. Since many people visit FOTF.com during this season to make a year-end gift, the “wiki-banner” serves as a low-friction tactic to get people to the donation page quickly. It is important to note that the visiting audience to FOTF’s website is served different content / experiences based on pages they have consumed in the past…specifically a “marriage audience” and an “in favor of pro-life audience – Santicity of Human Life” (also known as SOHL).
For this specific experiment targeting the “marriage audience”, we were focused on changing the language on the gift array buttons to reflect the impact that a gift would make on the amount of marriages saved. The theory was that if we could directly show how many marriages could be saved with a specific gift amount, the visitor would be moved to give a bigger gift to make a greater impact. As screen real-estate is at a premium for this “wiki-banner”, we designed an experience that showed the number of marriages saved on a button on the wiki-banner, and in a “hover-state” it would flip to the equivalent dollar amount of the gift. We even added a free-form field that would allow you to add how many marriages you wanted to save, and it would calculate that amount in real-time. After making a selection, the user is directed to the donation form where the amount they chose is auto-populated on the giving form, to reduce friction. It is also important to note that we were testing for revenue lift on desktop only (not mobile due to screen size limitations).
Research Question
We believe that changing the language on the gift array of a direct ask pop-up to show the impact of the gift will result in greater revenue
Design
Results
| Treatment Name | Revenue per Visitor | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | $1.14 | $169.50 | ||
| T1: | Marriage Language | $3.82 | 236.0% | 99.3% | $515.77 |
This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was met so these experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 10.4% increase in conversion rate
× 204.3% increase in average gift
236.0% increase in revenue
Key Learnings
If you notice, this experiment is labeled V1. We launched this pre-Giving Tuesday (11/23/22)….and were seeing very positive results…that is until the experiment was “over-ridden” inadvertently by FOTF due to some internal development directives. The over-ride happened right before Giving Tuesday, so we stopped getting results and ended up manually stopping the experiment on Dec 2, 2022. Prior to that stoppage, we recorded 236% lift in revenue for the treatment version with a 99.2% Level of Confidence. By statistical measures, this was a valid lift. Even when we removed outlier donations over $2k from both the control and the treatment, we still saw a significant lift from the treatment.
Due to the technical over-ride, we did not feel comfortable relying on these results to make any future recommendations other than to re-build the experiment as it was, and quickly re-launch to see if the results continued to show a positive lift in revenue through the remainder of the year-end campaign. Results from V2 will be shown in a future write-up and linked here.