How a static ad performs against a dynamic set of ads
Buckner International
Buckner International is a global ministry dedicated to the transformation and restoration of the lives we serve. We are a Christ-centered organization that delivers redemptive ministry to the most vulnerable from the beginning to the ending of life.
Experiment Summary
Timeframe: 04/29/2019 - 05/06/2019
In theory, Facebook’s algorithm has the potential to pair the perfect ad based upon an individual user’s preferences. To do this, Facebook created a style of ad known ad “Dynamic Ads.” In this situation, the marketer will create individual components for each ad that Facebook would then mix and match to create the best ad based upon each individual person. However, there is a learning curve that is often hard to overcome when working with small audience sizes or limited budgets. With Buckner’s current offer, Discovering God’s Will for your Life, we wanted to see if a single static ad that we felt was the “best” ad would outperform Facebook’s algorithm using Dynamic ads.
Research Question
Will a single static ad outperform Facebook’s dynamic ads?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Dynamic Ads | 0.00% | ||
| T1: | Static Ads | 0.09% | 100.0% | 98.6% |
This experiment has a required sample size of 4,093 in order to be valid. Since the experiment had a total sample size of 14,239, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 100.0% increase in conversion rate
× 0% increase in average gift
Key Learnings
After running for a week, we observed a statistically significant increase in donor acquisition using the static ads. With that said, it is worth noting that the number of conversions is lower than we would want to see before turning off an experiment. However, we also live in the real world. We do not want to keep spending advertising budget on a tactic that has produced a cost per donor that is double what we are seeing with the treatment.
With these two observations in mind, we decided to shut off the experiment and roll out the treatment to all audiences. There is also potential to re-test this concept on future campaigns.