How aligning the subject matter of a Facebook ad with the audience affects email acquisition rate Experiment ID: #6793

Hoover Institution

Experiment Summary

Timeframe: 7/13/2016 - 6/10/2017

The Hoover Institution was promoting their daily policy rundown newsletter, the Hoover Daily Report on Facebook to several audiences that had a variety of policy interests, including the economy, foreign policy, and many more. They were curious to see if focusing on one of these issues would result in an increase in email acquisition when tested against a general Hoover-focused ad. They created two ad variants to focus on the two commonly shared policy interests across the test audiences: one focusing on economic policy and one on foreign policy.

They launched a three-way test to determine which ad led to the best conversion rate.

Research Question

Will a single-issue ad increase conversion over a general newsletter offer?

Design

C: Hoover
T1: Economic
T2: Foreign Policy

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Hoover 0.70%
T1: Economic 0.54% -23.6% 100.0%
T2: Foreign Policy 0.77% 9.7% 98.6%

This experiment has a required sample size of 14,384 in order to be valid. Since the experiment had a total sample size of 521,617, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 9.7% increase in conversion rate
× 0% increase in average gift

Key Learnings

The two ad treatments saw a wide variance in results. The economic policy-focused ad produced a 23.6% decrease in conversions, while the foreign policy-focused ad produced a 9.7% increase over the control. This does not necessarily indicate that a foreign policy-focused ad is the best treatment for Hoover Daily Report as a whole. However, it does show that with these target audiences, a foreign policy focus results in the best cost-per-acquisition. This result prompted further testing across various audiences—especially ones that specialized in foreign policy and economics—to see if the results hold true.


Get the Detailed Case Study

Get a copy of the slides that highlight the different elements of the experiment, the variables that were changed, and the key takeaways and learnings that you can apply to your own organization.


Pick your format of choice:

Experiment Documented by...

Jeff Giddens

Jeff is the President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.