How increasing the page length and complicating the message affected donor conversion
Harvest Ministries
Harvest Christian Fellowship exists to bring Christians closer to God and to bring nonbelievers to a saving relationship with Him by showing how God's Word and faith in Him are applicable and relevant to everyday life.
Experiment Summary
Timeframe: 06/01/2016 - 06/07/2016
Harvest Ministries promoted their midmonth premium offer for the month which was, Thoughts to Make Your Heart Sing by Sally Lloyd-Jones. The email pushed people to the landing page to get the premium.
We hypothesized that if we added additional supportive organization-focused value proposition language we might be able to motivate more donors and increase conversion rate.
We created a treatment and ran an A/B test to determine which would perform better.
Research Question
Will increasing the force of the value proposition increase donor conversion rate?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 7.7% | ||
| T1: | Treatment 1 | 5.7% | -26.4% | 98.5% |
This experiment has a required sample size of 1,158 in order to be valid. Since the experiment had a total sample size of 3,572, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 26.4% decrease in conversion rate
× 0% increase in average gift
Key Learnings
While we hypothesized that the treatment with the additional value proposition copy would help improve donor conversion, it actually had a 26.4% decrease in donations. The added copy for the treatment made the page longer and it also introduced an additional value proposition which may have complicated the message – in turn creating friction to the giving process. Additional testing could be done to try and increase the force of the value proposition by reinforcing the value proposition about the premium itself.