How page formatting impacts engagement Experiment ID: #18370

Alliance Defending Freedom

Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.

Experiment Summary

Timeframe: 9/15/2019 - 10/1/2019

Alliance Defending Freedom had turned their blog posts into a significant source of new subscribers for the organization. As a result, much of the optimization efforts had been put into increasing that acquisition or the downstream conversion. However, during a discussion with the whole team, a theory was developed that if we could increase engagement on the page, we could potentially increase email acquisition too.

We decided to shift our efforts from direct email acquisition to indirect acquisition through increased engagement. One of the first efforts we decided to test was making the blog more readable. To do this, we increased the default font size and added quotes throughout the articles. Where relevant, we also added sub-headlines to make the text more scannable to the average reader.

Research Question

Will increasing the readability of the articles increase overall visitor engagement?


C: Control
T1: Updated Format


Treatment Name Open Rate Relative Difference Confidence
C: Control 62.6%
T1: Updated Format 63.8% 1.9% 100.0%

This experiment has a required sample size of 12,738 in order to be valid. Since the experiment had a total sample size of 95,202, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    1.9% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

While not a huge percentage, by simply increasing the font-size and adding elements to the page to make it easier to scan, we were able to increase the number of people to read the articles by 1.9%. (This was determined by measuring how many people made it 75% down the page.) Despite the small percentage, this means that roughly 2.5K more people engaged with the content than otherwise would have.

We also saw several other metrics move with this change:

  • There was a directional 7% increase in email acquisition (at an 80* level of confidence)
  • 3% increase in read rate when we excluded the most popular article that made up 72% of traffic

Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.