Americans for Prosperity

What happens when we use the candidate’s name on the donation page?

Experiment ID: #9465

Americans for Prosperity

Experiment Summary

Timeframe: 09/28/2018 - 10/04/2018

As the SCOTUS confirmation hearings for Brett Kavanaugh neared a completion, we wanted to test what would happen if we carried the candidate’s name through to the donation landing page that was sitting behind the Open Letter petition signature page. This hypothesis was based upon a learning we had on the previous page where we found the specificity of the candidate name increased email acquisition.

Research Question

What happens when we use the candidate’s name on the donation page instead of using a title to identify the candidate?

Design

C: Control
T1: Kavanaugh-specific

Results

 Treatment NameConv. RateRelative DifferenceConfidence
C: Control 0.38%
T1: Kavanaugh-specific 0.57%51.7% 97.0%

This experiment has a required sample size of 9,511 in order to be valid. Since the experiment had a total sample size of 23,271, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 51.7% increase in conversion rate
× 0% increase in average gift

Key Learnings

We found that incorporating the candidate name in the subsequent donation page had a 52% increase to donations. We have found that donation pages that follow open letters / petitions are often the hardest to see statistically significant lifts to giving. This is because their instant donor conversion rates are so low that it is difficult to validate. That is just one reason this result was so significant.

When first creating the control page, we tried to keep the language general so that donors would be giving to the organization and not to the candidate. We believed that the overarching principles of the organization would be more powerful than this single situation. However, this experiment reinforces that importance of clearly aligning the donation call-to-action to the experience they are coming from.


Experiment Documented by NextAfter

Question about experiment #9465

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.