How aligning the subject matter of a Facebook ad with the audience affects email acquisition rate
Hoover Institution
Experiment Summary
Timeframe: 07/13/2016 - 06/10/2017
The Hoover Institution was promoting their daily policy rundown newsletter, the Hoover Daily Report on Facebook to several audiences that had a variety of policy interests, including the economy, foreign policy, and many more. They were curious to see if focusing on one of these issues would result in an increase in email acquisition when tested against a general Hoover-focused ad. They created two ad variants to focus on the two commonly shared policy interests across the test audiences: one focusing on economic policy and one on foreign policy.
They launched a three-way test to determine which ad led to the best conversion rate.
Research Question
Will a single-issue ad increase conversion over a general newsletter offer?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Hoover | 0.70% | ||
| T1: | Economic | 0.54% | -23.6% | 100.0% |
| T2: | Foreign Policy | 0.77% | 9.7% | 98.6% |
This experiment has a required sample size of 14,384 in order to be valid. Since the experiment had a total sample size of 521,617, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 23.6% decrease in conversion rate
× 0% increase in average gift
Key Learnings
The two ad treatments saw a wide variance in results. The economic policy-focused ad produced a 23.6% decrease in conversions, while the foreign policy-focused ad produced a 9.7% increase over the control. This does not necessarily indicate that a foreign policy-focused ad is the best treatment for Hoover Daily Report as a whole. However, it does show that with these target audiences, a foreign policy focus results in the best cost-per-acquisition. This result prompted further testing across various audiences—especially ones that specialized in foreign policy and economics—to see if the results hold true.