How a radical donation page treatment impacts acquisition revenue Experiment ID: #17352

Buckner International

Buckner International is a global ministry dedicated to the transformation and restoration of the lives we serve. We are a Christ-centered organization that delivers redemptive ministry to the most vulnerable from the beginning to the ending of life.

Experiment Summary

Timeframe: 7/1/2019 - 8/3/2019

After running for nearly six months, we had not been able to achieve a statistically significant lift to the donation page used with Buckner’s 7 Scripture offer. After multiple iterative approaches, we decided to test a radical redesign of the page. We combined multiple factors based upon positive results seen with similar organizations.

The down side to this approach is that it makes it difficult to isolate exactly what helped move the needle if a lift is achieved. However, the overall goal was to lift revenue and if that could be achieved then knowing the exact reason for the lift became less important.

Research Question

Will a radical redesign of the donation page increase overall revenue to the acquisition campaign?

Design

C: Control
T1: Courtney's Version

Results

Treatment Name Revenue per Visitor Relative Difference Confidence Average Gift
C: Control $1.09 $24.70
T1: Courtney's Version $1.23 12.8% 74.3% $28.52

This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was not met so these experiment results are not valid.

Key Learnings

While we did not reach a statistical level of significance, we did see an increased average gift from the treatment version. Initial results saw the treatment also have a higher conversion rate but this balanced out after a few weeks of running. This is a good reminder of why it is so critical to ensure there is both a significant amount of time and total conversions before making a call on any experiment.

Our current hypothesis for the increase in average gift is that the treatment makes a targeted ask of $50 while the control version asks for $25. However, given all of the other factors that changed, we will need to isolate this factor to prove it out in the future.


Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.