iPad image vs Physical book image
KCBI
Experiment Summary
Ended On: 01/10/2020
Research Question
Will making the image of an eBook an iPad increase email signups?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Book image | 3.1% | ||
| T1: | iPad image | 2.3% | -26.5% | 100.0% |
This experiment has a required sample size of 3,166 in order to be valid. Since the experiment had a total sample size of 28,904, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 26.5% decrease in conversion rate
× 0% increase in average gift
Key Learnings
It does seem like Facebook may have favored one ad over the other. In the future I would like to run this same test from the beginning and potentially create a treatment landing page which also includes the iPad image in order to increase consistency for the user.