Texas State Historical Association

How testing into a new offer affects name acquisition

Experiment ID: #11702

Texas State Historical Association

Experiment Summary

Timeframe: 06/01/2015 - 06/05/2015

To take advantage of their high website traffic, the Texas State Historical Association has been offering a free eBook version of their popular 2014-2015 Texas Almanac in exchange for visitor email addresses. This name acquisition offer has been running for the past 6 months and continues to be a high performer. However, with the release of a recent History Channel show, the motivation of website visitors had shifted and we wanted to see if a different offer could further improve the acquisition efforts.

As a result, the TSHA ran a three way split on the traffic offering three different eBook options, the Texas Almanac, The Battle of the Alamo, and Remember Goliad! We wanted to see which of these offers could collect the most new email addresses for the organization.

Research Question

Which name acquisition offer produces the most new email addresses for the file?

Design

C: 2014-15 Texas Almanac eBook
T1: The Battle of the Alamo eBook
T2: Remember Goliad! eBook

Results

 Treatment NameConv. RateRelative DifferenceConfidence
C: 2014-15 Texas Almanac eBook 1.3%
T1: The Battle of the Alamo eBook 1.6%23.6% 99.7%
T2: Remember Goliad! eBook 0.99%-21.1% 99.7%

This experiment has a required sample size of 4,564 in order to be valid. Since the experiment had a total sample size of 85,932, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 23.6% increase in conversion rate
× 0% increase in average gift

Key Learnings

The Alamo eBook improved email acquisition by 23.6% while maintaining the immediate donor conversion on the follow up screen. This increased volume of email addresses will allow for additional conversion opportunities down the line.

As a result, the TSHA will be changing offers to just focus on the Alamo eBook for the next several weeks. Once the show is over, we will want to re-validate these results to ensure the motivation (and corresponding conversion) does not change.


Experiment Documented by NextAfter

Question about experiment #11702

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.