How the deliver method for a content offer can significantly impact acquisition.
Care Net
Experiment Summary
Timeframe: 02/01/2018 - 02/21/2018
On the Care Net site, they offer a couple of content offers to people coming to their site. This is a great way to acquire emails using their site traffic. The offers are currently being delivered through exit-intent pop-ups. We hypothesized that we might be able to increase email conversion if we used a different delivery method. We proposed testing a slide-out feature on their site.
Research Question
Would a slide-out feature increase email conversion? How would it impact specific audiences and devices?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Exit-intent | 0.88% | ||
| T1: | Slide-out | 3.5% | 296.4% | 100.0% |
This experiment has a required sample size of 243 in order to be valid. Since the experiment had a total sample size of 6,878, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 296.4% increase in conversion rate
× 0% increase in average gift
Key Learnings
By presenting the content offers with a slide out vs an exit-intent pop-up, we were able to increase email acquisition by 296%! We found that the treatment was more impactful on individual audiences. Specifically, on mobile and traffic visitors, we saw the slide out have a statistically significant 870% lift on email acquisition. On the flip side, the desktop version had a 184% lift. We wanted to dig into these individual audiences because the experiences for the Control version varied by device; this is due to an exit intent not working on a mobile phone. In those situations, they would see a banner at the top of the screen as they scrolled down. Given this situation, the significant increase makes more sense due to the disruptive nature of the slide out when compared to a small banner at the bottom.
We will re-evaluate this tactic a few months from now. This type of acquisition can begin to decrease in effectiveness as visitors get used to seeing it. We want to make sure we are maximizing the return from organic website visitors.