How promoting a video, then retargeting video viewers with an offer decreased results
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Timeframe: 10/13/2019 - 11/10/2019
Facebook agency partners consistently tell agencies, who then tell the clients they serve, that the best way to promote an offer is to launch with a video, optimize your campaign for video views, then retarget video viewers with an offer.
The Heritage Foundation decided that they would like to test this process once and for all and get to the bottom of whether or not this approach does indeed increase results.
Research Question
Would targeting prospecting audiences with a related video to a content offer increase donor acquisition campaign results?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Purchase Optimized Content Offer | 0.01% | ||
| T1: | Video View » Purchase Optimized Content Offer | 0.00% | -98.5% | 100.0% |
This experiment has a required sample size of 11,654 in order to be valid. Since the experiment had a total sample size of 1,174,073, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 98.5% decrease in conversion rate
× 0% increase in average gift
Key Learnings
With a 99.9% level of confidence, we observed a decrease in donor conversion rate of -98.5% when targeting prospecting audiences with a video, followed by a content offer promotion.
With that said, here are some positive observations from the treatment experience:
- Our audience “reach” (the number of unique people that saw our ad at least once) increased by 198%.
- Our impressions increased by 332%.
However, these are the other decreases for the treatment experience that we witnessed during this experiment:
- Clicks decreased by 67%.
- The number of emails we captured decreased by 86%.
- The number of donors we created decreased by 98%.
- Revenue decreased by 95%
These results were produced on a slightly less (-6%) investment relative to what was spent on the control experience.
Where everything falls apart for this experience type is in the cost per result or outcome measurements. Here is what we observed:
- Our cost per email acquired increased nearly 6x over the control experience.
- Our cost to acquire a donor increased by slightly more than 13x over the control experience.
- Our ROI on this type of campaign decreased by 95.5% when compared to results achieved in the control experience.
Key Learnings:
- For nonprofit organizations who need to squeeze the best returns from every dollar invested, it would appear that video promotion + offer promotion to video viewers is absolutely NOT a viable path to grow your file and maximize returns to the organization on ad spend.
- If your organization is attempting to grow awareness (and perhaps purchase intent or favorability on a particular issue), video optimized campaigns certainly appear to be effective in terms of generating reach, however brand lift experimentation is advised to be sure of the exact value from that type of investment.
- Purchase optimized content offers still appear to be the best option when it comes to producing a tangible return on investment (emails, donors, instant revenue).