How Facebook’s optimization event impacts donor conversion rate
Americans for Prosperity
Experiment Summary
Ended On: 04/01/2021
In anticipation of iOS 14 privacy updates that were forthcoming in Spring of 2021, Americans for Prosperity wanted to experiment with understanding whether the optimization event they had historically used (called “Conversions”) was better than the alternative (called “Value”).
If the “Value” optimization event improved results, it would determine how they proceeded with domain verification and pixel tracking setup ahead of the new iOS 14 that was about to be launched.
How would the “value” optimization event compare to their standard “conversion” event type?
Research Question
How would the “value” optimization event compare to their standard “conversion” event type?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.06% | ||
| T1: | Treatment #1 | 0.03% | -50.5% | 99.9% |
This experiment has a required sample size of 38,573 in order to be valid. Since the experiment had a total sample size of 232,987, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 50.5% decrease in conversion rate
× 0% increase in average gift
Key Learnings
With a 99.9% level of confidence, we observed that the “value” conversion event in Facebook decreased donor conversion rate by -50.5%.
Subsequently, we also observed a decrease of revenue by 53% (LoC: 99.9%).
This is actually very good news, as the “value” conversion event was going to dramatically increase the complexity of iOS 14 setup and event tracking prioritization. Therefore, this decrease was very welcomed.
Where possible, prioritize your Facebook pixel event tracking towards “conversions” as it appears that this event appears to be more effective at driving overall donors, and subsequently revenue.