How the delivery of the offer can impact email acquisition Experiment ID: #6266

Illinois Policy Institute

Experiment Summary

Timeframe: 2/15/2017 - 3/8/2017

Illinois Policy Institute’s website sees significant volumes of daily traffic. They also have a weekly newsletter that contains investigative reporting and other top headlines from around the state of Illinois. Historically, this newsletter could be subscribed to by clicking on a link found in the footer of every page.

We had the hypothesis that we could increase email acquisition by making the offer easier to find by brining it up higher in the page. We decided to test two new methods to see which would have the greatest impact, an inline email acquisition offer or a slide out that appeared as a visitor scrolled down the page.

Research Question

Which email offer would produce the highest email acquisition rate?


C: No Direct Offer
T1: Inline Newsletter Offer
T2: Slideout Newsletter Offer


Treatment Name Conv. Rate Relative Difference Confidence
C: No Direct Offer 0.14%
T1: Inline Newsletter Offer 0.61% 350.1% 100.0%
T2: Slideout Newsletter Offer 0.78% 470.2% 100.0%

This experiment has a required sample size of 1,422 in order to be valid. Since the experiment had a total sample size of 68,404, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 470.2% increase in conversion rate
× 0% increase in average gift

Key Learnings

Both the inline email acquisition and slide out were able to lift the email acquisition of the site. The slide out offer proved to be the most effective with a 470% lift. Our hypothesis is that this was due to the offer being more disruptive to the reading process which arrested the visitors’ attention.

Get the Detailed Case Study

Get a copy of the slides that highlight the different elements of the experiment, the variables that were changed, and the key takeaways and learnings that you can apply to your own organization.

Pick your format of choice:

Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.