How an ‘Up Next’ slideout affects email acquisition rate on Desktop

Alliance Defending Freedom

Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.

Experiment Summary

Timeframe: 09/05/2017 - 10/02/2017

Alliance Defending Freedom recently conducted an experiment  on mobile devices, where they asked: Is it possible to increase clickthrough rate by providing an ‘Up Next’ article? The result was a valid 29% increase in traffic without a valid decrease in conversion. They wondered if they would see the same results if they conducted this experiment with desktop users as well. They launched a desktop-only test to find out.

Research Question

Can we increase clickthrough rate on desktop by providing a single article that is “Up Next”?

Design

C: No Up Next Slidout

T1: Has Up Next Slideout

Results

This experiment has a required sample size of 70,000 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

Alliance Defending Freedom’s experiment did not achieve validity. Though the results were inconclusive, this test may tell us an important thing or two about desktop users. Behavior from one device to another can change significantly. We know that those who view a blog post on a mobile device are already more likely to read more than one article at a time. Due to the differences in motivation and the likely environment of mobile and desktop users, it’s important that we test each separately. Another variable that makes this test different than the mobile-only version is the ‘Up Next’ content the desktop viewer saw. It’s possible this piece of content was less compelling to viewers than what was shown as a treatment in the mobile test.

Further studies are being conducted to determine whether desktop visitors benefit from an ‘Up Next’ slideout or not.