iPad image vs Physical book image Experiment ID: #19515

KCBI

Experiment Summary

Ended On: 1/10/2020

As part of an advertising campaign for their new eBook offer, KCBI wanted to test the impact of different ways of visualizing it. Since it was digital, we had the hypothesis that an iPad would be a more clear visual for what they would receiving but we wanted to test that against the standard book image that is often used with these visuals.

Research Question

Will making the image of an eBook an iPad increase email signups?

Design

C: Book image
T1: iPad image

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Book image 3.1%
T1: iPad image 2.3% -26.5% 100.0%

This experiment has a required sample size of 3,166 in order to be valid. Since the experiment had a total sample size of 28,904, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 26.5% decrease in conversion rate
× 0% increase in average gift

Key Learnings

It does seem like the Facebook audience found the image of an actual book more appealing which resulted in a greater number of clicks and, subsequently, more emails acquired. One possible hypothesis for this is that the age of the audience (largely 65+ adults) likely resonated more with seeing the physical book rather than the iPad image of it. It is also possible that this is a one-off result and should be further tested with additional offers.


Experiment Documented by...

Allan Torres

Allan is Optimization Associate. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.