Focus on the Family

How showing an abandoned donation pop-up to users based on entry point affects revenue – homepage version

Experiment ID: #114068

Focus on the Family

Focus on the Family is a global Christian ministry dedicated to helping families thrive. We provide help and resources for couples to build healthy marriages that reflect God's design, and for parents to raise their children according to morals and values grounded in biblical principles.

Experiment Summary

Ended On: 11/11/2022

Focus on the Family wanted to see if showing a pop-up with an opportunity to complete a donation (from a previously clicked on amount), would change based on the entry point of a visitor. The thought was that user intentions might be different for visitors coming directly to the home page vs visitors entering from other pages.

Prior to this experiment, we first tested if showing a pop-up to visitors (regardless of entry point) would increase overall completed transactions. We built this initial test with a single pop up that was shown to all visitors that visited a donation page, clicked on an amount to give, but abandoned the process after that. The pop-up would show to that visitor as they navigated around on the FOTF site.

For this test, we split the test into two separate tests. One test focussed on visitors to all non-homepages. The pop up was shown only to folks that came through anywhere but the main page, hit the donate button, clicked an amount…then abandoned the process.

This particular test focused on visitors to the main home page. The pop up behaved exactly the same, except that it only showed to people that visited through the homepage page first. The idea was that people coming through main page would be more familiar and thus more likely to complete a transaction, but they may need more reminders if they happened to go through the donation pathway…and possibly the abandoned donation pop up would serve as that gentle nudge and cause a completed transaction, and thus more revenue.

Research Question

We believe that visitors entering the site on the main page will be prompted to complete their donation when presented a pop-up reminder and we will generate more revenue than not showing a pop-up.

Design

C: Control
T1: Abandoned Donation Pop Up - Homepage

Results

 Treatment NameRevenue per VisitorRelative DifferenceConfidenceAverage Gift
C: Control $1.25$70.69
T1: Abandoned Donation Pop Up - Homepage $1.11-11.1% 53.3%$68.78

This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was not met so these experiment results are not valid.

Key Learnings

By showing a pop-up reminder to complete the transaction, our theory was that those that came to the main page to make a donation, but abandoned for some reason, would potentially be swayed by this gentle nudge. Just like the test on the non-homepage version, When we pulled transaction data, we found quite a few gifts that skewed the results of both the control and the treatment. We decided to remove any donation of $2k or more from both the treatment and the control in order to remove any outliers that might be skewing the results. When we did this, we found the data to be inconclusive with an 11% decrease in the treatment, but with only a 53% Level of Confidence. There is no real positive or negative effect on the results. It is also important to note that experiments for FOTF that run outside of high-urgency time frames behave much differently than during campaign periods.

The key takeaway is that when running transaction experiments that involve revenue, we cannot take the results on face value. We must dig into the numbers deeper, look for any anomalies that can skew results, and then evaluate for lift. In this case, while we did not see a real difference either way (after removing outlier data), we also did not see a potentially harmful outcome. While we cannot recommend that this current variation become the new control, we can recommend to keep testing different variations of the same experiment. Next up is a reduced ask (60% of the originally clicked donation amount), but only on the non-homepage version.


Experiment Documented by NextAfter

Question about experiment #114068

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.