How page formatting impacts engagement
Alliance Defending Freedom
Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.
Experiment Summary
Timeframe: 09/15/2019 - 10/01/2019
Alliance Defending Freedom had turned their blog posts into a significant source of new subscribers for the organization. As a result, much of the optimization efforts had been put into increasing that acquisition or the downstream conversion. However, during a discussion with the whole team, a theory was developed that if we could increase engagement on the page, we could potentially increase email acquisition too.
We decided to shift our efforts from direct email acquisition to indirect acquisition through increased engagement. One of the first efforts we decided to test was making the blog more readable. To do this, we increased the default font size and added quotes throughout the articles. Where relevant, we also added sub-headlines to make the text more scannable to the average reader.
Research Question
Will increasing the readability of the articles increase overall visitor engagement?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 62.6% | ||
| T1: | Updated Format | 63.8% | 1.9% | 100.0% |
This experiment has a required sample size of 12,738 in order to be valid. Since the experiment had a total sample size of 95,202, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
1.9% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
While not a huge percentage, by simply increasing the font-size and adding elements to the page to make it easier to scan, we were able to increase the number of people to read the articles by 1.9%. (This was determined by measuring how many people made it 75% down the page.) Despite the small percentage, this means that roughly 2.5K more people engaged with the content than otherwise would have.
We also saw several other metrics move with this change:
- There was a directional 7% increase in email acquisition (at an 80* level of confidence)
- 3% increase in read rate when we excluded the most popular article that made up 72% of traffic