First Baptist Dallas

Does using Yes/NO ads to promote a new premium offer impact donor conversion rates?

Experiment ID: #163499

First Baptist Dallas

Experiment Summary

Timeframe: 10/20/2023 - 11/06/2023

Pathway to Victory is the broadcast ministry arm of First Baptist Dallas, one of the largest Baptist churches in the country.

For a new resource that was created to respond to questions PTV had been getting with the chaos in Israel, we wanted to test whether using yes/no creative would make an impact on donor conversions. The yes/no concept is something PTV had used on organic posts; we had not tested this before for Facebook ads.

In this experiment, which we ran as adding an additional creative element to a campaign, not a straight A/B test, we tested the control which was an image of the book cover against the yes/no ads.

Research Question

We believe that using yes/no ads for a Facebook audience will achieve a change in conversion rates.


C: Control
T1: Treatment #1


 Treatment NameConv. RateRelative DifferenceConfidence
C: Control 0.04%
T1: Treatment #1 0.03%-16.0% 49.1%

This experiment has a required sample size of 821,684 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

The key learning from this experiment is that using yes/no ads for a Facebook audience did not result in a significant change in conversion rates. The treatment actually saw a decrease in donations compared to the control group.

It is important to note that the statistical level of confidence for this experiment is below 85%, which means the results cannot be considered valid. Therefore, these results should not be used for future decision making.

While CPMs across both creative variations were the same, there was a slight decrease in CTR for the variant (1.09% compared to 0.93%). However the cost to acquire a donor on the yes/no variant was 16% higher.

In future experimentation, it would be beneficial to conduct further tests using different ad formats, messaging, or targeting strategies to see if there are any other variables that could impact conversion rates. Additionally, it may be worth exploring other channels or platforms to reach the target audience. Experimentation is crucial for understanding what strategies work best for increasing conversions, and it is important to evaluate results with a statistically significant level of confidence before making any conclusive decisions.

Experiment Documented by NextAfter

Question about experiment #163499

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.