How 3rd party credibility impacts the effectiveness of an ad

Americans for Prosperity

Experiment Summary

Timeframe: 07/01/2018 - 07/10/2018

American for Prosperity was running an Open Letter campaign to acquire signatures for their letter to the Senate regarding the upcoming Supreme Court nominee. The primary source of traffic was targeted ads running on Facebook. AFP has a unique value proposition in that they have access to 3.2 million activists around the country. It was our hypothesis that this large number of supporters could serve as a 3rd party credibility indicator which would reduce the anxiety associated with signing an open letter. We created multiple versions to test this theory.

Research Question

Will the 3.2 million activists serve as a 3rd party credibility indicator which will help increase email acquisition?

Design

C: Control

T1: Shortened Version

T2: 3rd Party Credibility

Results

This experiment has a required sample size of 4,942 in order to be valid. Since the experiment had a total sample size of 139,793, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 16.3% increase in conversion rate
× 0% increase in average gift

Key Learnings

We found that the ad that included the “3.2 million other Americans” stat was able to increase email acquisition by 16.3%. The other treatment was basically a shortened version of the Control ad. With this second treatment (the short version), we were able to also disprove the hypothesis that the 3rd Party Credibility treatment won simply because it was shorter.

Knowing the impact of the “3.2 million other Americans” has on acquisition, it is a fact that we will continue to use in future acquisition efforts.