How a copy-heavy popup ad affects email acquisition
NextAfter
Experiment Summary
Timeframe: 12/17/2020 - 12/20/2020
In this experiment, we were running a pop-up ad with a nonprofit marketing partner. For the sake of time, we whipped up version A with minimal copy and a very simple design.
But once the campaign launched, we wondered if running a very text-heavy ad would actually lead to emails since it communicated the value far more effectively.
We assumed we would get lower clicks, but we thought the clicks we would get would be more qualified – thus leading to greater conversion.
Research Question
We believe that a more value heavy ad for Top Nonprofits visitors will achieve more email acquisition.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.26% | ||
| T1: | Treatment #1 | 0.26% | 0.0% | 0.0% |
This experiment has a required sample size of 1 in order to be valid. Since the experiment had a total sample size of 8,500, and the level of confidence is not above 95% the experiment results are not valid.
Key Learnings
We saw the exact same amount of conversion from both ads. As predicted, version B had significant decrease in clicks (158 VS 67), but a much higher conversion rate for those that viewed the landing page (6.6% VS 15%).
In this case, the drop in clicks and increase in conversion rate meant that our overall acquisition was a wash.
This indicates to me that there may be a happy medium between these two ad variations. It would be worth testing a new ad treatment with some value prop copy to help qualify the click, but leave more to be desired to discover on the next page.