How including the “plan” details in an ad impacted email acquisition Experiment ID: #9268

Americans for Prosperity

Experiment Summary

Timeframe: 7/16/2018 - 7/23/2018

Americans for Prosperity was running an Open Letter as part of their campaign for the confirmation of Judge Kavanaugh. The primary source of traffic was visitors driven by Facebook ads. In a previous experiment, we found that including the “3.2 million other Americans” in the ad increased the effectiveness of the offer. As a follow up experiment, we wanted to see if adding even more context to the size and scope of the effort would further boost results.

Research Question

Will adding additional details of the “plan” to the Facebook ad increase overall email acquisition rate?


C: Control
T1: "Overwhelming voice"


Treatment Name Conv. Rate Relative Difference Confidence
C: Control 3.3%
T1: "Overwhelming voice" 4.7% 44.2% 100.0%

This experiment has a required sample size of 1,604 in order to be valid. Since the experiment had a total sample size of 375,536, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 44.2% increase in conversion rate
× 0% increase in average gift

Key Learnings

By including information on how the Open Letter fit into the overall plan for confirmation increased email acquisition by 44%. There are several suggested “best practices” that would suggest Facebook ads should be short and to the point. This experiment is another example that proves if we can provide additional information that increases the perceived value of the action, it will increase overall results, regardless of the ad length.

Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.

Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.