Focus on the Family

How showing an abandoned donation pop-up to users based on entry point affects revenue – non-homepage version

Experiment ID: #113221

Focus on the Family

Experiment Summary

Ended On: 11/03/2022

Focus on the Family wanted to see if showing a pop-up with an opportunity to complete a donation (from a previously clicked on amount), would change based on the entry point of a visitor. The thought was that user intentions might be different for visitors coming directly to the home page vs visitors entering from other pages.

Prior to this experiment, we first tested if showing a pop-up to visitors (regardless of entry point) would increase overall completed transactions. We built this initial test with a single pop up that was shown to all visitors that visited a donation page, clicked on an amount to give, but abandoned the process after that. The pop-up would show to that visitor as they navigated around on the FOTF site.

For this test, we split the test into two separate tests. One test focussed on visitors to the main page. The pop up was shown only to folks that came through the main page, hit the donate button, clicked an amount…then abandoned the process. This particular test focused on visitors to all other pages. The pop up behaved exactly the same, except that it only showed to people that visited through non-homepage pages first. The idea was that people coming through main page would be more familiar and thus more likey to complete a transaction, where as visitors that came to article pages or via a SEO search, may need more reminders if they happened to go through the donation pathway…and possibly the abandoned donation pop up would serve as that gentle nudge and cause a completed transaction, and thus more revenue.

Research Question

We believe that visitors entering the site outside of the main page will be prompted to complete their donation when presented a pop-up reminder and we will generate more revenue than not showing a pop-up.

Design

C: Control

T1: Treatment #1

Results

	Treatment Name	Revenue per Visitor	Relative Difference	Confidence	Average Gift
C:	Control	$0.24			$72.34
T1:	Treatment #1	$0.00	-100.0%	0.0%	$0.00

This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was not met so these experiment results are not valid.

Key Learnings

This test provided some interesting early data. Our hypothesis was that motivations for people visiting the site outside of the main page would be lower to donate than those from main site visits. By showing a pop-up reminder to complete the transaction, our theory was that those that had less motivation would potentially be swayed by this gentle nudge. After about 10 days of running the experiment, the test actually validated (with a greater than 95% LoC) based on revenue. The test results also showed that average gift for the treatment was substantially higher (over $25 greater for the treatment). While these initial results were very encouraging, whenever we run a test on revenue lift, we always need to look for outlier donations that could potentially skew the data results. When we pulled transaction data, we found quite a few gifts that skewed the results of both the control and the treatment. There was a $20k gift on the control, and several $5k plus gifts on the treatment. We decided to remove any donation of $1k or more from both the treatment and the control in order to remove any outliers that might be skewing the results. When we did this, we found the data to be inconclusive. There is no real positive or negative effect on the results. It is also important to note that experiments for FOTF that run outside of high-urgency time frames behave much differently than during campaign periods.

The key takeaway is that when running transaction experiments that involve revenue, we cannot take the results on face value. We must dig into the numbers deeper, look for any anomalies that can skew results, and then evaluate for lift. In this case, while we did not see a real difference either way (after removing outlier data), we also did not see a potentially harmful outcome. While we cannot recommend that this current variation become the new control, we can recommend to keep testing different variations of the same experiment. Next up is a reduced ask (60% of the originally clicked donation amount).

Experiment Documented by NextAfter

Question about experiment #113221

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.