How a Facebook headline affects direct-ask donor conversion
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Timeframe: 10/24/2017 - 12/12/2017
The Heritage Foundation noticed an opportunity on Facebook to retarget visitors who viewed their online donation pages. They created to direct ask Facebook ads that invited anyone who visited the page but did not complete the form to return and support the Heritage Foundation. In the first ad, they simply asked all visitors (including members who visited the page) to ‘Donate now.’ In the second ad, the asked all non-member visitors to join the Heritage Foundation by ‘Becoming a member today.’ They launched a test to find out which approach would be more effective at converting donors.
Research Question
Will the headline ‘Donate now’ or ‘Become a member’ drive more donor conversions on a Facebook ad?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.65% | ||
| T1: | Treatment 1 | 0.00% | -100.0% | 68.4% |
This experiment has a required sample size of 676 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
They found that that the results were inconclusive. Due to a low overall donor conversion rate, The Heritage Foundation decided to pause this campaign and assess the results. The first ad did receive 1 donation, but this was not enough to validate this approach over the other. Using the urgency of the calendar year, They decided to launch and run a similar test that is directly tied to their CYE theme.