Billy Graham Evangelistic Association

How increasing perceived security on a primary donation page for mobile devices can impact donor conversion

Experiment ID: #13030

Billy Graham Evangelistic Association

Experiment Summary

Timeframe: 02/26/2019 - 03/26/2019

One of the most common reasons web visitors don’t complete transactions online is a perceived lack of security. Knowing this, Billy Graham Evangelical Association wondered if their primary donation pathway was losing potential donors because it didn’t visually represent how secure it was. They created a special security box around the credit card section of form and added a padlock into the shaded box to represent that the form was secure. This didn’t make the form more secure—it was already PCI compliant and had the highest levels of security. All these design tweaks did was increase perception of security with the end user. They launched an A/B test on all device types (desktop/mobile/tablet) to see if it increased donor conversion as they hypothesized.

Research Question

Will adding visual elements increase the perception of form security increase donor conversion on mobile devices?

Design

C: Control
T1: Treatment #1

Results

 Treatment NameConv. RateRelative DifferenceConfidence
C: Control 39.2%
T1: Treatment #1 32.8%-16.3% 100.0%

This experiment has a required sample size of 431 in order to be valid. Since the experiment had a total sample size of 4,609, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 16.3% decrease in conversion rate
× 0% increase in average gift

Key Learnings

Adding visual elements to increase the perception of security on the donation form decreased donor conversion by 16% on mobile devices. We hypothesize that this decrease could be in the way that the visual elements look on mobile. Instead of reducing friction for the donor pathway on mobile devices, the visual security elements may have actually increased this by making the credit card section stand out from the rest of the elements on the form, drawing attention to it being called out. 

Even though we saw a 16% decrease in donor conversion on mobile devices, we saw a 20% increase in donor conversion on desktop devices. While it is important to look at the impact on all devices, it is important to know that the majority of donations and revenue (67%) on the primary donation page comes from desktop devices. Because the revenue impact is so much greater on desktop than on other devices, we recommend rolling out this tactic on desktop to continue to increase donations and revenue, while no longer featuring the tactic on mobile to eliminate the negative impact.  


Experiment Documented by Courtney Gaines
Courtney Gaines is Vice President at NextAfter.

Question about experiment #13030

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.