How referring to the author’s name versus their famous work generates more clicks
Hillsdale College
Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.
Experiment Summary
Timeframe: 04/05/2018 - 04/19/2018
Hillsdale College offers courses on literary author Jane Austen. This test was conducted in the subject line. We wanted to know if mentioning Jane Austen’s name in the subject line and alluding to her life before becoming an author would cause a different open rate than the subject line about her work before her most famous book.
Research Question
Does referring to the roots of the author’s “genius” generate more conversions than the history of the author’s works?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | The roots of Jane Austen's genius | 1.1% | ||
| T1: | Before there was Pride and Prejudice | 0.91% | -15.7% | 99.9% |
This experiment has a required sample size of 26,415 in order to be valid. Since the experiment had a total sample size of 152,322, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
15.7% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The control had a click rate of 1.1%, compared to a click rate of 0.91% of the treatment. Since there was not a significant difference in the click rate between the two versions, we can conclude that the different concepts had no effect on opens/clicks.
Note: the copy in each email is different, but both pieces of copy follow the same idea as their respective subject lines.