How 3rd party credibility impacts the effectiveness of an ad Experiment ID: #9247

Americans for Prosperity

Experiment Summary

Timeframe: 7/1/2018 - 7/10/2018

American for Prosperity was running an Open Letter campaign to acquire signatures for their letter to the Senate regarding the upcoming Supreme Court nominee. The primary source of traffic was targeted ads running on Facebook. AFP has a unique value proposition in that they have access to 3.2 million activists around the country. It was our hypothesis that this large number of supporters could serve as a 3rd party credibility indicator which would reduce the anxiety associated with signing an open letter. We created multiple versions to test this theory.

Research Question

Will the 3.2 million activists serve as a 3rd party credibility indicator which will help increase email acquisition?

Design

C: Control
T1: Shortened Version
T2: 3rd Party Credibility

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Control 4.9%
T1: Shortened Version 4.6% -5.6% 91.1%
T2: 3rd Party Credibility 5.7% 16.3% 100.0%

This experiment has a required sample size of 4,942 in order to be valid. Since the experiment had a total sample size of 139,793, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 16.3% increase in conversion rate
× 0% increase in average gift

Key Learnings

We found that the ad that included the “3.2 million other Americans” stat was able to increase email acquisition by 16.3%. The other treatment was basically a shortened version of the Control ad. With this second treatment (the short version), we were able to also disprove the hypothesis that the 3rd Party Credibility treatment won simply because it was shorter.

Knowing the impact of the “3.2 million other Americans” has on acquisition, it is a fact that we will continue to use in future acquisition efforts.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.