How creating a “shortcut” to the donation form affects conversion

Heritage Action for America

Experiment Summary

Timeframe: 09/09/2015 - 09/17/2015

Leading up to their Republican presidential debates, Heritage Action for America ran a money bomb campaign to create more awareness around the event. This campaign was a two week long series with multiple emails all driving to the same landing page.

Previous experiments had proven that longer copy is better at communicating the value proposition to the donor.  However, we had a hypothesis that we may be able to create a shortcut for some donors that did not need as much convincing. The idea was to put a “Donate Now” button in the top right of the page that would jump the potential donors to the donation form at the bottom thus reducing friction.

Research Question

How will removing the “Donate Now” button on a money bomb impact the number of donations and revenue?

Design

C: No Donate Now Button

T1: Donate Now Button

Results

This experiment has a required sample size of 2,632 in order to be valid. Since the experiment had a total sample size of 5,470, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 28.2% decrease in conversion rate
× 34.1% decrease in average gift
52.6% decrease in revenue

Key Learnings

By adding a “Donate Now” button to the top right corner of the page, we ended up reducing the number of donations by 28%. Our hypothesis is that this button prematurely tipped our hand on the purpose of the campaign and made the ask before the visitors were ready to see it. While the long page copy may add friction to the giving process, the added benefit that comes with a strong value proposition is enough to overcome that friction.