How creating a congruent experience impacted visitor engagement Experiment ID: #5625

Alliance Defending Freedom

Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.

Experiment Summary

Timeframe: 10/15/2016 - 11/7/2016

Alliance Defending Freedom maintains a blog section of their website that is updated at least once per day. A recent data analysis discovered that, while there was significant volume to the blog, the engagement of this audience was low. On average 85% of all traffic left within the first 10 seconds of arriving on the site.

After digging deeper into the website metrics, we discovered that mobile and desktop traffic were suffering the most from low engagement. Since the primary goal of the blog is to engage visitors, we decided to run an experiment designed to get more visitors to stay longer on the site. Our primary measure of success would be the number of visitors that stayed on the site longer than 60 seconds.

Our hypothesis was that there was a disconnect between the environment that the majority of visitors were coming from (in this case, Facebook) and the environment they were landing on. As a result, we decided to test out a mobile blog experience that was more congruent with the Facebook’s style of browsing.

We stripped out many of the images, reduced the navigation and secondary calls-to-action, and then gave visitors a small amount of the content with the remainder hidden behind a “Read More” button.  The goal of the “Read More” button was to expose them to the additional content found at the bottom of the page.

Research Question

Will a mobile experience that is more congruent with the visitors’ original source (Facebook) increase their engagement with the content?


C: Original Blog Design
T1: Mobile Optimized Blog Design


Treatment Name Click Rate Relative Difference Confidence
C: Original Blog Design 11.1%
T1: Mobile Optimized Blog Design 19.5% 75.6% 100.0%

This experiment has a required sample size of 142 in order to be valid. Since the experiment had a total sample size of 38,339, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    75.6% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The mobile-optimized blog yielded a 76% increase in the number of visitors that stayed on the site longer than 60 seconds. This equated to 1,656 more people that consumed ADF’s content in a 3 week period.

While the 60-second mark was the primary success metric for this experiment, looking at some of the earlier times yielded even larger lifts in engagement. We saw a 182% increase in the number of visitors that stayed at least 30 seconds and a 366% increase in the number that stayed longer than 10 seconds.

To us, this demonstrates the importance of visual congruency for the visitor. If we are able to match the environment they are coming from, the visitors will be more likely to engage with and consume the content.

Get the Detailed Case Study

Get a copy of the slides that highlight the different elements of the experiment, the variables that were changed, and the key takeaways and learnings that you can apply to your own organization.

Pick your format of choice:

Related Experiments

This experiment is part of a series of experiments aimed at improving overall results. Take a look at some of the other iterations:

Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.