Does using Yes/NO ads to promote a new premium offer impact donor conversion rates?
First Baptist Dallas
Experiment Summary
Timeframe: 10/20/2023 - 11/06/2023
Pathway to Victory is the broadcast ministry arm of First Baptist Dallas, one of the largest Baptist churches in the country.
For a new resource that was created to respond to questions PTV had been getting with the chaos in Israel, we wanted to test whether using yes/no creative would make an impact on donor conversions. The yes/no concept is something PTV had used on organic posts; we had not tested this before for Facebook ads.
In this experiment, which we ran as adding an additional creative element to a campaign, not a straight A/B test, we tested the control which was an image of the book cover against the yes/no ads.
Research Question
We believe that using yes/no ads for a Facebook audience will achieve a change in conversion rates.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.04% | ||
| T1: | Treatment #1 | 0.03% | -16.0% | 49.1% |
This experiment has a required sample size of 821,684 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
The key learning from this experiment is that using yes/no ads for a Facebook audience did not result in a significant change in conversion rates. The treatment actually saw a decrease in donations compared to the control group.
It is important to note that the statistical level of confidence for this experiment is below 85%, which means the results cannot be considered valid. Therefore, these results should not be used for future decision making.
While CPMs across both creative variations were the same, there was a slight decrease in CTR for the variant (1.09% compared to 0.93%). However the cost to acquire a donor on the yes/no variant was 16% higher.
In future experimentation, it would be beneficial to conduct further tests using different ad formats, messaging, or targeting strategies to see if there are any other variables that could impact conversion rates. Additionally, it may be worth exploring other channels or platforms to reach the target audience. Experimentation is crucial for understanding what strategies work best for increasing conversions, and it is important to evaluate results with a statistically significant level of confidence before making any conclusive decisions.