How presenting an email acquisition opportunity as a forced pop-up impacts conversion Experiment ID: #16969

Billy Graham Evangelistic Association

Experiment Summary

Timeframe: 4/1/2019 - 6/1/2019

Billy Graham Evangelical Association gets significant traffic to their Answers pages. At the bottom of these pages there is an opportunity for people to sign up to receive the BGEA newsletter. We hypothesized that presenting that acquisition opportunity in a different way might increase conversion and name acquisition for the organization. To test this, we developed a pop-up/exit-intent pop-up that appeared on desktop (and a mobile slide-up). Knowing that a lot of the people coming to the Answers pages were returning visitors, and that a forced pop-up on a page can cause a poor user experience, we delayed the pop-up from appearing until after a person had been on the page for 30 seconds. If they exited prior to that time and seeing the pop-up, we would then trigger an exit-intent pop-up. In addition to this, we also put a rule on the pop-ups to only show every third visit.

Research Question

Can we increase name acquisition on the Answers pages through a forced and exit-intent pop-up. 


C: Control
T1: Popup


Treatment Name Conv. Rate Relative Difference Confidence Average Gift
C: Control 0.36% $0.00
T1: Popup 0.70% 93.3% 100.0% $0.00

This experiment has a required sample size of 3,559 in order to be valid. Since the experiment had a total sample size of 858,709, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 93.3% increase in conversion rate
× 0% increase in average gift

Key Learnings

The new method for presenting the acquisition opportunity increased conversion by 93.3%! The newsletter sign-up on the bottom of the page was getting lost with all of the other content on the page. By pulling that offer up from the bottom and presenting it when people were more highly engaged with the site, we were able to increase the motivation of people on the page to get the offer. In addition, the trigger parameters that were put on the pop-up created a more positive user experience and will allow the pop-up to be effective longer than just using a forced pop-up with no rules in place. 

In our next phase of testing, we would like to test the kind of offer in the pop-up to see if we can increase the relevancy and appeal of what is being offered to people on these high-trafficked, highly-engaged pages.

Experiment Documented by...

Courtney Gaines

Courtney is the Senior Director of Optimization at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.