FamilyLife

How the placement of a Dear Listener message vs. an Impact Pop-up on a donation page impacts donor conversion during a year end campaign

Experiment ID: #194503

FamilyLife

Experiment Summary

Timeframe: 12/02/2023 - 12/16/2023

FamilyLife, a nonprofit dedicated to strengthening families through biblically-based resources, ran this experiment as part of their high urgency calendar year-end campaign, which began on December 2nd. Historically, FamilyLife has relied on a “Dear Listener” tactic at the end of their content—an approach that appeals directly to their audience and has proven successful in driving donations during previous high urgency campaigns. For this test, however, we sought to explore whether a different tactic could outperform this traditional method in terms of donor conversion.

To determine whether a more interactive and urgent format would yield higher conversions, we tested the Impact Pop-up against the “Dear Listener” message. The “Dear Listener” message was used in the Control group, maintaining the traditional approach where the appeal is made at the end of the content. In the Treatment group, we implemented the Impact Pop-up, which was triggered during the user’s session and emphasized the urgency of supporting the mission. The pop-up also featured an image of a family reading a Bible to reinforce the message and visually engage users.

This experiment was set up as a split A/B test in VWO, with the traffic being split 50/50 between the Control (Dear Listener) and the Treatment (Impact Pop-up). We tested these tactics on their podcast pages. Both versions used similar messaging, but the key differences were the format and the visibility of the appeal.

Research Question

We believe that using the Impact Pop-up for FamilyLife’s calendar year-end campaign will increase donor conversion better than the Dear Listener message because the pop-up creates a more immediate sense of urgency and engagement, encouraging users to take action right away.

Design

C: Control: Dear Listener

T1: Treatment: Impact Pop-up

Results

	Treatment Name	Conv. Rate	Relative Difference	Confidence
C:	Control: Dear Listener	0.28%
T1:	Treatment: Impact Pop-up	0.43%	51.8%	67.7%

This experiment has a required sample size of 12,752 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

The experiment ran from December 2nd to December 16th, a two-week period, and although it did not reach the ideal 30-day threshold for full validation, the Impact Pop-up showed a promising 51.8% increase in conversions with a 67.6% likelihood of conversion (LoC) compared to the Dear Listener message. The Dear Listener generated 9 transactions, while the Impact Pop-up generated 14 transactions. While the result is compelling, it was still a bit too early to validate fully, but given the urgency of the campaign, the decision was made to roll out the Impact Pop-up. The Impact Pop-up likely performed better because it created a sense of urgency by appearing during the user’s session, prompting immediate action. Unlike the Dear Listener message, which required users to scroll down the page to see it, the Impact Pop-up was triggered after 7 seconds of page engagement, ensuring that most visitors to the page would see it. This guaranteed visibility likely contributed to higher engagement and conversion, as users didn’t need to actively seek out the message. The emotional appeal of the image of a family reading the Bible further reinforced the urgency, making the call-to-action feel more relevant and personal for users. Given the success of the Impact Pop-up in this initial period, we recommend rolling it out as the primary tactic for future high urgency campaigns. However, since the experiment didn’t run long enough to validate fully, we suggest retesting, continuing to monitor its performance closely, and conducting follow-up testing to further optimize the pop-up’s design and messaging. Additionally, testing with a larger sample size and longer duration could help validate its effectiveness across different user segments.

Experiment Documented by NextAfter

Question about experiment #194503

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.