How a 3rd party endorsement affects email acquisition

Texas State Historical Association

Experiment Summary

Timeframe: 02/09/2016 - 02/15/2016

After the initial launch of the Texas State Historical Association’s eBook celebrating Black History Month, they received an endorsement for the work from a well known public figure, the mayor of Houston. The quote promoted both TSHA and the eBook itself. We had the hypothesis that if we added the quote to the page, it would decrease the anxiety which would, in turn, increase the conversion rate. We decided to test this out against the existing control page.

Research Question

Does adding a 3rd party endorsement improve the email acquisition rate on the page?

Design

C: No Quote

T1: Mayor Quote

Results

This experiment has a required sample size of 1,034 in order to be valid. Since the experiment had a total sample size of 4,148, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 28.0% decrease in conversion rate
× 0% increase in average gift

Key Learnings

The 3rd party quote ended up decreasing the email acquisition rate on the page by 28%. We have two hypotheses for the cause of this decrease:

1. The audience has come to expect that the products released by TSHA are of high quality. As a result, they didn’t need the added 3rd party credibility. Instead, this quote added length to the page which added friction to the conversion process.
2. It is possible that the endorser, the Houston mayor, is unpopular with the audience demographics. Regardless of their reasonings, if the visitors did not like the Houston mayor they would be less likely to convert.

In either situation, it made sense to remove the quote and continue with the original page.