How removing video and simplifying the donation form affect conversion - NextAfter
Luther Seminary

How removing video and simplifying the donation form affect conversion

Experiment ID: #76095

Luther Seminary

Experiment Summary

Timeframe: 11/10/2021 - 11/25/2021

Luther Seminary was running their biannual Working Preacher donation campaign. The campaign had traditionally used the main WP donation page, with a special video and their standard donation page.

But based on previous research, the team wanted to test the impact of making two significant changes. First, they wanted to test the impact of removing the video, as video had shown to reduce conversion for other campaigns.

Secondly, they wanted to see how simplifying the donation process and reordering the steps would affect conversion. As it stood, the donation form had a thermometer, a gift array, and then asked for a comment. It made a recurring ask after the gift amount was chosen, which could lead to cognitive friction between the previously selected amount and the monthly ask. If someone selected a $100 amount, would they want to give that monthly? They wanted to test asking for the frequency of gift first, then following up with the amount. They also made the gift array decrease when monthly ask was selected to align with traditional monthly ask amounts. Finally, they removed the comments box, which presented unnecessary friction.

To understand the unique impact of these changes, they ran an A/B/C test where variant B only removed the video, and variant C removed the video and added the “tabbed” simplified form.

Research Question

We believe that removing the video and reordering and simplifying the donation process for website visitors will achieve an increase in conversion rate.

Design

C: Control
T1: Remove video
T2: Remove video / tabbed form

Results

  Treatment Name Conv. Rate Relative Difference Confidence Average Gift
C: Control 34.9% $0.00
T1: Remove video 40.0% 14.5% 44.8% $0.00
T2: Remove video / tabbed form 53.1% 52.0% 97.4% $0.00

This experiment has a required sample size of 86 in order to be valid. Since the experiment had a total sample size of 209, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 52.0% increase in conversion rate
× 0% increase in average gift

Key Learnings

Treatment C, with the removed video and the reordered, simplified donation form produced a 52% increase in conversion rate, with 97.3% significance. Digging deeper into the data, they found two interesting facts that showed how this was derived.

First, email traffic (which has a higher motivation due to reading a promotional message about the campaign) was largely unchanged. But direct traffic, the source of the most donations, was where the 52% lift was found.

Secondly, these changes did not affect desktop traffic at all, looking at all channels. But mobile traffic skyrocketed, with a 177.6% increase with a 99% level of confidence.

So in short, large gains in direct traffic (which has lower motivations) and mobile traffic (which has device-specific restrictions and where a video takes over the entire screen) were the sources of lift for this winning treatment.


Experiment Documented by Jeff Giddens
Jeff Giddens is President of NextAfter.

Question about experiment #76095

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.