How relevancy and recency impact visitor engagement
Alliance Defending Freedom
Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.
Experiment Summary
Timeframe: 11/18/2016 - 11/30/2016
Alliance Defending Freedom had recently made a radical shift in their blog to better engage their mobile visitors. They created an entirely separate website that would mirror the experience most mobile visitors were coming from (Facebook). This change significantly increased engagement for the high volume of traffic that visit their blog each day.
The next iteration in this test series was to see if we could get the visitors to explore the ADF website in order to learn more about the organization. One hypothesis was that if we showed the visitors content that was more relevant to the post they had just read, they would be more likely to continue exploring the site.
To test that hypothesis, we created a version of the blog that replaced the most recent blog posts with related content.
Research Question
Will related content drive more engagement and exploration than recent content?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Recent Content | 3.9% | ||
| T1: | Related Content | 3.1% | -20.3% | 95.3% |
This experiment has a required sample size of 4,192 in order to be valid. Since the experiment had a total sample size of 8,569, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
20.3% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The related content ended up decreasing the number of visitors that read other articles by 20%. This was a big discovery since there was significant effort being put into making this a feature available on the desktop version of the blog. This will allow ADF to focus their efforts on website improvements that can drive deeper engagement.