How removing a single click affects email acquisition
Harvest Ministries
Harvest Christian Fellowship exists to bring Christians closer to God and to bring nonbelievers to a saving relationship with Him by showing how God's Word and faith in Him are applicable and relevant to everyday life.
Experiment Summary
Timeframe: 03/24/2016 - 04/01/2016
Harvest Ministries received a lot of traffic to their main homepage. In one of the boxes on the page, they offered a sample of one of Pastor Greg Laurie’s daily devotionals. After the sample, there was a button that triggered a modal window to receive the daily devotions by email.
They hypothesized that allowing the visitor to sign up for the devotionals directly in the daily devotion box would reduce friction and increase the amount of people who signed up.
They launched an A/B test to determine a result.
Research Question
Will removing a single click reduce friction and increase email acquisition rate?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Modal Popup | 0.20% | ||
| T1: | Inline Signup | 0.42% | 113.3% | 100.0% |
This experiment has a required sample size of 5,352 in order to be valid. Since the experiment had a total sample size of 42,560, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 113.3% increase in conversion rate
× 0% increase in average gift
Key Learnings
This shows the immense power of reducing very small amounts of friction. Asking the visitors to click to subscribe meant that a significant percentage of potential subscribers never even saw the form.
By removing that single action, Harvest was able to significantly increase the number of visitors who accepted the offer.