How the cover image of an ebook increased results Experiment ID: #18699

The Fund for American Studies

Experiment Summary

Timeframe: 10/23/2019 - 11/8/2019

The Fund for American Studies launched a new acquisition offer and after running it for a month and a half, the team needed to improve ad clickthrough rate and email signup rates to boost performance for the offer.

Research Question

Would changing the ebook cover design image used in ads and on the landing page improve results?

Design

C: LFS - Cover Test (Control)
T1: LFS - Cover Test (Treatment)

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: LFS - Cover Test (Control) 1.9%
T1: LFS - Cover Test (Treatment) 3.1% 65.1% 100.0%

This experiment has a required sample size of 1,227 in order to be valid. Since the experiment had a total sample size of 21,678, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 65.1% increase in conversion rate
× 0% increase in average gift

Key Learnings

With a level of confidence of 99.9%, we validated an increase of 65.1% in email signup rate for this treatment.

This was partially driven by an increase of +25.9% in ad clickthrough rate (LoC: 99.2%). We also saw a directional lift in donor conversion rate by +6.4% (LoC: 9.8%).

The landing page results increased because the motivation of those arriving at the page, combined with the higher number of people (because more ads were clicked) increased results substantially.


Experiment Documented by...

Greg Colunga

Greg is Executive Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.