How placement of a sticky bar affects email acquisition on mobile devices
Leadership Institute
Experiment Summary
Ended On: 11/15/2018
Leadership Institute sought to acquire as many email addresses as possible from their flagship news site, CampusReform.org. They had put email acquisition offers on the site through “sticky bars” and sought to optimize both through testing and optimization on both desktop and mobile. They had tested to a sticky bar that appeared when someone scrolled through 25% of the page at the bottom of the page. They hypothesized that this might be interrupting the user experience too much on mobile, as the content removed their ability to scroll down. They decided to test placement—from the current “bottom anchored” bar to the “top anchored” bar.
Research Question
How will a sticky bar anchored to the top of the mobile experience perform against one anchored to the bottom?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Lower Sticky Bar | 2.3% | ||
| T1: | Upper Sticky Bar | 4.9% | 110.3% | 96.2% |
This experiment has a required sample size of 434 in order to be valid. Since the experiment had a total sample size of 873, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 110.3% increase in conversion rate
× 0% increase in average gift
Key Learnings
The top-anchored sticky bar produced a 110% lift in name acquisition than the bottom-anchored sticky bar—which showed that where you place the sticky bar in terms of user experience has a tremendous impact on the results it produces.
Ultimately, this test established a new standard—future testing should focus on design and value proposition, but this provides a helpful framework for understanding user experience.