How the deliver method for a content offer can significantly impact acquisition. Experiment ID: #8573

Care Net

Experiment Summary

Timeframe: 2/1/2018 - 2/21/2018

On the Care Net site, they offer a couple of content offers to people coming to their site. This is a great way to acquire emails using their site traffic. The offers are currently being delivered through exit-intent pop-ups. We hypothesized that we might be able to increase email conversion if we used a different delivery method. We proposed testing a slide-out feature on their site.

Research Question

Would a slide-out feature increase email conversion? How would it impact specific audiences and devices?

Design

C: Exit-intent
T1: Slide-out

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Exit-intent 0.88%
T1: Slide-out 3.5% 296.4% 100.0%

This experiment has a required sample size of 243 in order to be valid. Since the experiment had a total sample size of 6,878, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 296.4% increase in conversion rate
× 0% increase in average gift

Key Learnings

By presenting the content offers with a slide out vs an exit-intent pop-up, we were able to increase email acquisition by 296%! We found that the treatment was more impactful on individual audiences. Specifically, on mobile and traffic visitors, we saw the slide out have a statistically significant 870% lift on email acquisition. On the flip side, the desktop version had a 184% lift. We wanted to dig into these individual audiences because the experiences for the Control version varied by device; this is due to an exit intent not working on a mobile phone. In those situations, they would see a banner at the top of the screen as they scrolled down. Given this situation, the significant increase makes more sense due to the disruptive nature of the slide out when compared to a small banner at the bottom.

We will re-evaluate this tactic a few months from now. This type of acquisition can begin to decrease in effectiveness as visitors get used to seeing it. We want to make sure we are maximizing the return from organic website visitors.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.