How A/B testing was able to find an error buried in NetCommunity
Buckner International
Buckner International is a global ministry dedicated to the transformation and restoration of the lives we serve. We are a Christ-centered organization that delivers redemptive ministry to the most vulnerable from the beginning to the ending of life.
Experiment Summary
Timeframe: 09/15/2016 - 09/18/2016
When we initially launched Buckner International’s latest offer, a free copy of a book written by Buckner president Dr. Albert L. Reyes, we saw that the landing page had a lower than expected conversion rate. We were able to drive significant traffic through Facebook ads but were collecting 60% less emails than we expected.
After evaluating the landing page, we had the hypothesis that the problem may not be with the offer but with the system itself. We had recently created a new template for Buckner’s NetCommunity instance that we were worried may not work for some people. We decided to launch the same landing page created on a different system (Unbounce) as a way to validate our hypothesis. If all things were working as expected, we would expect to see no significant difference between the two systems.
Research Question
Is there any issue that is being caused by the system hosting the landing page (NetCommunity)?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | NetCommunity | 19.1% | ||
| T1: | Unbounce | 37.3% | 95.2% | 100.0% |
This experiment has a required sample size of 46 in order to be valid. Since the experiment had a total sample size of 325, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 95.2% increase in conversion rate
× 0% increase in average gift
Key Learnings
After 3 days of running, we saw that Unbounce had a 95% higher conversion rate than the same NetCommunity page. This implied that there was an issue with the system. Using this information, we dug deeper into Google Analytics and discovered that traffic coming to the site on an Android device seemed to have the biggest gap in conversion rate. This was a device that we hadn’t specifically tested on so when we loaded the page, we saw this screenshot:
This was being caused by a rogue file that had been included on that page that linked to an unsecure server. No other browser seemed to be throwing this error which is why finding it was so difficult. We have since been able to remove the reference to the file which has corrected the issue.