How creating a congruent experience impacted visitor engagement
Alliance Defending Freedom
Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.
Experiment Summary
Timeframe: 10/15/2016 - 11/07/2016
Alliance Defending Freedom maintains a blog section of their website that is updated at least once per day. A recent data analysis discovered that, while there was significant volume to the blog, the engagement of this audience was low. On average 85% of all traffic left within the first 10 seconds of arriving on the site.
After digging deeper into the website metrics, we discovered that mobile and desktop traffic were suffering the most from low engagement. Since the primary goal of the blog is to engage visitors, we decided to run an experiment designed to get more visitors to stay longer on the site. Our primary measure of success would be the number of visitors that stayed on the site longer than 60 seconds.
Our hypothesis was that there was a disconnect between the environment that the majority of visitors were coming from (in this case, Facebook) and the environment they were landing on. As a result, we decided to test out a mobile blog experience that was more congruent with the Facebook’s style of browsing.
We stripped out many of the images, reduced the navigation and secondary calls-to-action, and then gave visitors a small amount of the content with the remainder hidden behind a “Read More” button. The goal of the “Read More” button was to expose them to the additional content found at the bottom of the page.
Research Question
Will a mobile experience that is more congruent with the visitors’ original source (Facebook) increase their engagement with the content?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Original Blog Design | 11.1% | ||
| T1: | Mobile Optimized Blog Design | 19.5% | 75.6% | 100.0% |
This experiment has a required sample size of 142 in order to be valid. Since the experiment had a total sample size of 38,339, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
75.6% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The mobile-optimized blog yielded a 76% increase in the number of visitors that stayed on the site longer than 60 seconds. This equated to 1,656 more people that consumed ADF’s content in a 3 week period.
While the 60-second mark was the primary success metric for this experiment, looking at some of the earlier times yielded even larger lifts in engagement. We saw a 182% increase in the number of visitors that stayed at least 30 seconds and a 366% increase in the number that stayed longer than 10 seconds.
To us, this demonstrates the importance of visual congruency for the visitor. If we are able to match the environment they are coming from, the visitors will be more likely to engage with and consume the content.