How a survey question in a Facebook Ad impacted email signup rates
NextAfter
Experiment Summary
Ended On: 10/22/2022
As a part of our own marketing and prospecting via Facebook, we decided to try a new survey-style ad image to see if we could generate a much higher volume of ad clicks, and subsequently from that extra traffic, convert more people on the subsequent landing page.
Research Question
Our hypothesis is that by using a survey-style ad (with a “Yes/No” question included in the ad image) will increase clicks on the ad so much, that we can generate more emails at a more efficient cost.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.29% | ||
| T1: | Treatment #1 | 0.12% | -57.8% | 100.0% |
This experiment has a required sample size of 5,441 in order to be valid. Since the experiment had a total sample size of 52,520, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 57.8% decrease in conversion rate
× 0% increase in average gift
Key Learnings
With a confidence level of 100%, we observed a decrease in email signups by -57.8%.
Our hypothesis completely failed with our audiences on Facebook Advertising, because this decrease in email signups was actually driven by fewer clicks overall (we saw that this treatment ad decreased ad clicks by -53.6%, also with a confidence level of 100%).
Ultimately, we’d like to re-try this experiment, but next time with a different question in the treatment ad. With evidence that these style of ads work well in other cases, the thought here is that perhaps the question is too vague. Furthermore, could the length of the question also be impacting ad clickthrough rates?
Would asking a more simple question (like: “Do you want to get more from your Facebook advertising?”) get straight to the point, and resonate with a larger portion of the audience? It’s worth testing, so we will soon.