Scouts

How a delayed pop-up affects email acquisition rate

Experiment ID: #93149

Scouts

Experiment Summary

Timeframe: 05/02/2022 - 05/22/2022

We built a National Scouting survey pop-up modal to live on the BSA homepage to drive survey completions. Internal teams were concerned with site UX and asked that we set pop-up to fire after the user was on the page for 15 seconds. We believed a shorter duration of 5 seconds would produce a better result and so we created a split test for half of the users to see a pop-up at 15 seconds and the other half to receive it after 5 seconds had elapsed.

Research Question

We believe that triggering a pop-up in a shorter duration for site visitors will achieve a better survey completion/email capture rate.

Design

C: Pop-up (5 secs)
T1: Pop-up (15 secs)

Results

 Treatment NameConv. RateRelative DifferenceConfidenceAverage Gift
C: Pop-up (5 secs) 0.26%
T1: Pop-up (15 secs) 0.11%-56.4% 99.9%

This experiment has a required sample size of 6,768 in order to be valid. Since the experiment had a total sample size of 41,336, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 56.4% decrease in conversion rate
× 0% increase in average gift

Key Learnings

The 5 second pop-up clearly outperformed waiting 15 seconds to trigger the modal. The 5 second variant acquired 53 emails versus the 23 acquired by the 15 second version. The 15 second version underperformed the 5 second version by 56% with a 99.9% confidence rate, proving that the shorted trigger duration was clearly the stronger performing variant.

Hypothesis for this performance has more to do with time on page and were users even on the page for 15 seconds to see the longer triggered variant on the page or had they already navigated elsewhere within the BSA site.

Follow up test will be to shorten the pop-up trigger from 5 seconds to 2 seconds. Test to follow this one will test trigger time versus triggering pop-up upon page scroll.


Experiment Documented by NextAfter

Question about experiment #93149

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.