How migrating systems can improve conversion rate
Buckner International
Buckner International is a global ministry dedicated to the transformation and restoration of the lives we serve. We are a Christ-centered organization that delivers redemptive ministry to the most vulnerable from the beginning to the ending of life.
Experiment Summary
Timeframe: 02/20/2018 - 03/10/2018
Buckner International was in the final stages of migrating from one email and landing page system to another. Before moving the email acquisition campaign entirely, we wanted to verify that there were no unforeseen issues or technical bugs that would pop up. To do this, we created an A/A experiment in which both pages were nearly identical; the only major difference was the system used to host the pages and send the emails. Our goal with this experiment was to demonstrate that there was no negative impact on conversion. Basically, with this kind of test, no statistical significance would have been considered a win.
Research Question
Will moving to a new system have any significant impact on email acquisition?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Unbounce / Mailchimp | 53.2% | ||
| T1: | Luminate | 61.3% | 15.3% | 99.4% |
This experiment has a required sample size of 284 in order to be valid. Since the experiment had a total sample size of 1,110, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 15.3% increase in conversion rate
× 0% increase in average gift
Key Learnings
After running for a full 10 days, we discovered that the new system was actually able to fix a bug that we didn’t know we had. It resulted in a 15% increase in conversion! We had seen comments in Facebook (which was t he main traffic source for this campaign) about people having trouble accessing the resource. After multiple tests, we were unable to reproduce any issue so we assumed it was a user error given the already high conversion rate. These results would indicate that the issue did exist and the new system was able to correctly address it.