How a delayed pop-up affects email acquisition rate
Scouts
Experiment Summary
Timeframe: 05/02/2022 - 05/22/2022
We built a National Scouting survey pop-up modal to live on the BSA homepage to drive survey completions. Internal teams were concerned with site UX and asked that we set pop-up to fire after the user was on the page for 15 seconds. We believed a shorter duration of 5 seconds would produce a better result and so we created a split test for half of the users to see a pop-up at 15 seconds and the other half to receive it after 5 seconds had elapsed.
Research Question
We believe that triggering a pop-up in a shorter duration for site visitors will achieve a better survey completion/email capture rate.
Design
Results
Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
---|---|---|---|---|---|
C: | Pop-up (5 secs) | 0.26% | |||
T1: | Pop-up (15 secs) | 0.11% | -56.4% | 99.9% |
This experiment has a required sample size of 6,768 in order to be valid. Since the experiment had a total sample size of 41,336, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 56.4% decrease in conversion rate
× 0% increase in average gift
Key Learnings
The 5 second pop-up clearly outperformed waiting 15 seconds to trigger the modal. The 5 second variant acquired 53 emails versus the 23 acquired by the 15 second version. The 15 second version underperformed the 5 second version by 56% with a 99.9% confidence rate, proving that the shorted trigger duration was clearly the stronger performing variant.
Hypothesis for this performance has more to do with time on page and were users even on the page for 15 seconds to see the longer triggered variant on the page or had they already navigated elsewhere within the BSA site.
Follow up test will be to shorten the pop-up trigger from 5 seconds to 2 seconds. Test to follow this one will test trigger time versus triggering pop-up upon page scroll.
Question about experiment #93149
If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.