How the style of Facebook ad optimization impacts results Experiment ID: #12122

Buckner International

Buckner International is a global ministry dedicated to the transformation and restoration of the lives we serve. We are a Christ-centered organization that delivers redemptive ministry to the most vulnerable from the beginning to the ending of life.

Experiment Summary

Timeframe: 2/16/2019 - 2/25/2019

Buckner International had been seeing considerable success with their long-running devotional email acquisition offer. Experiments geared toward conversion optimization and the donation value proposition had led to significant boosts in donor acquisition. Since we were seeing wins by testing some of the more technical aspects of Facebook advertising, we decided to continue this series of experiments.

Our next hypothesis was related to the method of budget optimization. There are multiple ways to allocate media spend within Facebook’s Ad Manager. We had previously tested setting a daily budget regardless of audience but that resulted in slight decrease in conversion rate. We wondered if removing the requirement for daily spending would help increase our results. To do this, we changed our budget optimization method to be based upon a Campaign Lifetime Budget.

Research Question

Will setting a Campaign Lifetime Budget increase results over setting Ad Set level budgets?

Design

C: Ad Set Budgets
T1: Campaign Lifetime Budget

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Ad Set Budgets 0.18%
T1: Campaign Lifetime Budget 0.05% -68.9% 99.2%

This experiment has a required sample size of 6,451 in order to be valid. Since the experiment had a total sample size of 21,074, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 68.9% decrease in conversion rate
× 0% increase in average gift

Key Learnings

After running for over a week, we found that the Campaign Lifetime Budget resulted in an 69% decrease in donor acquisition. In addition to the decrease in donors, we noticed that it spent 30% less than the control group. This means that even with a more controlled spend, the audience reached resulted in significantly lower results. 

Since we were seeing decreased results both in spend and donor acquisition, we decided to end this experiment and move on to the next experiment. This means we will need to continue our Ad Set level budgeting. It requires additional work to manage but the results are well worth the effort.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.