How the kind of content offer delivery mechanism impacts conversion rate Experiment ID: #8467

Care Net

Experiment Summary

Timeframe: 1/24/2018 - 2/1/2018

Care Net has a variety of content offers that they offer to people who come to their website. The offers were being delivered to people via an exit-intent pop-up on their site. This means that as a person would start to leave the page, a pop-up would appear on their screen with one of thecontent offers. We hypothesized that with the high volume of mobile traffic on their website, a different delivery mechanism might help increase name acquisition from their site since exit-intent pop-ups don’t work or show up on mobile.

Research Question

Which delivery mechanism would increase name acquisition?


C: Exit-intent
T1: Slideout


Treatment Name Conv. Rate Relative Difference Confidence
C: Exit-intent 1.1%
T1: Slideout 3.2% 194.8% 100.0%

This experiment has a required sample size of 371 in order to be valid. Since the experiment had a total sample size of 2,338, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 194.8% increase in conversion rate
× 0% increase in average gift

Key Learnings

After a little over a week, we saw a significant difference in conversion between the exit-intent pop-up and slide out. The slide-out increased conversion by 194.8%! After analyzing the data, we want to see the kind of impact this kind of experiment would have on desktop vs mobile. This would require us to run the experiment longer to get a larger sample size. The larger sample size would also help reinforce the significant impact and lift that this kind of delivery mechanism can have on acquisition as a whole.

Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.

Experiment Documented by...

Courtney Gaines

Courtney is the Senior Director of Optimization at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.