How A/B testing was able to find an error buried in NetCommunity Experiment ID: #4969

Buckner International

Buckner International is a global ministry dedicated to the transformation and restoration of the lives we serve. We are a Christ-centered organization that delivers redemptive ministry to the most vulnerable from the beginning to the ending of life.

Experiment Summary

Timeframe: 9/15/2016 - 9/18/2016

When we initially launched Buckner International’s latest offer, a free copy of a book written by Buckner president Dr. Albert L. Reyes, we saw that the landing page had a lower than expected conversion rate. We were able to drive significant traffic through Facebook ads but were collecting 60% less emails than we expected.

After evaluating the landing page, we had the hypothesis that the problem may not be with the offer but with the system itself. We had recently created a new template for Buckner’s NetCommunity instance that we were worried may not work for some people. We decided to launch the same landing page created on a different system (Unbounce) as a way to validate our hypothesis. If all things were working as expected, we would expect to see no significant difference between the two systems.

Research Question

Is there any issue that is being caused by the system hosting the landing page (NetCommunity)?

MECLABS Conversion Factors Targeted

C = 4m + 3v + 2( i - f) - 2a ©

Copyright 2015, MECLABS

Design

C: NetCommunity
T1: Unbounce

Results

Treatment Name Conv. Rate Relative Difference Confidence Average Gift
C: NetCommunity 19.1% $25.00
T1: Unbounce 37.3% 95.2% 100.0% $0.00

This experiment has a required sample size of 46 in order to be valid. Since the experiment had a total sample size of 325, and the level of confidence is above 95%, the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 95.2% increase in conversion rate
× 100.0% decrease in average gift
100.0% decrease in revenue

Key Learnings

After 3 days of running, we saw that Unbounce had a 95% higher conversion rate than the same NetCommunity page. This implied that there was an issue with the system. Using this information, we dug deeper into Google Analytics and discovered that traffic coming to the site on an Android device seemed to have the biggest gap in conversion rate. This was a device that we hadn’t specifically tested on so when we loaded the page, we saw this screenshot:

screenshot-2016-09-19-10-45-34

This was being caused by a rogue file that had been included on that page that linked to an unsecure server. No other browser seemed to be throwing this error which is why finding it was so difficult. We have since been able to remove the reference to the file which has corrected the issue.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Kevin Peters

Kevin is a Senior Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.