How presenting an email acquisition opportunity as a forced pop-up impacts conversion
Billy Graham Evangelistic Association
Experiment Summary
Timeframe: 04/01/2019 - 06/01/2019
Billy Graham Evangelical Association gets significant traffic to their Answers pages. At the bottom of these pages there is an opportunity for people to sign up to receive the BGEA newsletter. We hypothesized that presenting that acquisition opportunity in a different way might increase conversion and name acquisition for the organization. To test this, we developed a pop-up/exit-intent pop-up that appeared on desktop (and a mobile slide-up). Knowing that a lot of the people coming to the Answers pages were returning visitors, and that a forced pop-up on a page can cause a poor user experience, we delayed the pop-up from appearing until after a person had been on the page for 30 seconds. If they exited prior to that time and seeing the pop-up, we would then trigger an exit-intent pop-up. In addition to this, we also put a rule on the pop-ups to only show every third visit.
Research Question
Can we increase name acquisition on the Answers pages through a forced and exit-intent pop-up.
Design
Results
Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
---|---|---|---|---|---|
C: | Control | 0.36% | $0.00 | ||
T1: | Popup | 0.70% | 93.3% | 100.0% | $0.00 |
This experiment has a required sample size of 3,559 in order to be valid. Since the experiment had a total sample size of 858,709, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 93.3% increase in conversion rate
× 0% increase in average gift
Key Learnings
The new method for presenting the acquisition opportunity increased conversion by 93.3%! The newsletter sign-up on the bottom of the page was getting lost with all of the other content on the page. By pulling that offer up from the bottom and presenting it when people were more highly engaged with the site, we were able to increase the motivation of people on the page to get the offer. In addition, the trigger parameters that were put on the pop-up created a more positive user experience and will allow the pop-up to be effective longer than just using a forced pop-up with no rules in place.
In our next phase of testing, we would like to test the kind of offer in the pop-up to see if we can increase the relevancy and appeal of what is being offered to people on these high-trafficked, highly-engaged pages.
Question about experiment #11236
If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.