The Fund for American Studies

How increasing the clarity about the author impacted email signup rates for an eBook offer

Experiment ID: #35438

The Fund for American Studies

Experiment Summary

Timeframe: 08/25/2020 - 09/03/2020

As a part of the launch of a new eBook, The Fund for American Studies decided to experiment with increasing author clarity for the new resource they were promoting. Could this increase email signup rates, and subsequent donor conversion rates?

Research Question

How does increasing the author clarity of a new eBook offer impact email signup and donor conversion rates?

Design

C: No Author Clarity
T1: w/Author Clarity

Results

 Treatment NameConv. RateRelative DifferenceConfidence
C: No Author Clarity 61.4%
T1: w/Author Clarity 53.0%-13.7% 100.0%

This experiment has a required sample size of 266 in order to be valid. Since the experiment had a total sample size of 2,105, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 13.7% decrease in conversion rate
× 0% increase in average gift

Key Learnings

With a 100% level of confidence, we observed a decrease in email signup rate of -13.7% with the increased clarity for who the author is.

Subsequently, we not only observed fewer downloaders going through the treatment page, but with an 80% level of confidence (thereby making the observation not valid—yet, directionally accurate) we also observed a decrease in subsequent donor conversion by 29.9%, as well.

This is a baffling study. Typically, increasing the clarity of some component of the offer (in this case—who wrote it) will increase the strength of the value proposition, and subsequently increase results.

For that reason, we recommend further experimentation with this concept in terms of layout/design to continue testing this hypothesis to see if we can separate the author clarity from the presentation of the information itself.


Experiment Documented by Greg Colunga
Greg Colunga is Executive Vice President at NextAfter.

Question about experiment #35438

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.