How the style of Facebook ad optimization impacts results
Buckner International
Buckner International is a global ministry dedicated to the transformation and restoration of the lives we serve. We are a Christ-centered organization that delivers redemptive ministry to the most vulnerable from the beginning to the ending of life.
Experiment Summary
Timeframe: 02/16/2019 - 02/25/2019
Buckner International had been seeing considerable success with their long-running devotional email acquisition offer. Experiments geared toward conversion optimization and the donation value proposition had led to significant boosts in donor acquisition. Since we were seeing wins by testing some of the more technical aspects of Facebook advertising, we decided to continue this series of experiments.
Our next hypothesis was related to the method of budget optimization. There are multiple ways to allocate media spend within Facebook’s Ad Manager. We had previously tested setting a daily budget regardless of audience but that resulted in slight decrease in conversion rate. We wondered if removing the requirement for daily spending would help increase our results. To do this, we changed our budget optimization method to be based upon a Campaign Lifetime Budget.
Research Question
Will setting a Campaign Lifetime Budget increase results over setting Ad Set level budgets?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Ad Set Budgets | 0.18% | ||
| T1: | Campaign Lifetime Budget | 0.05% | -68.9% | 99.2% |
This experiment has a required sample size of 6,451 in order to be valid. Since the experiment had a total sample size of 21,074, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 68.9% decrease in conversion rate
× 0% increase in average gift
Key Learnings
After running for over a week, we found that the Campaign Lifetime Budget resulted in an 69% decrease in donor acquisition. In addition to the decrease in donors, we noticed that it spent 30% less than the control group. This means that even with a more controlled spend, the audience reached resulted in significantly lower results.
Since we were seeing decreased results both in spend and donor acquisition, we decided to end this experiment and move on to the next experiment. This means we will need to continue our Ad Set level budgeting. It requires additional work to manage but the results are well worth the effort.