Does personalization perform better than mystery in subject line?
Hillsdale College
Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.
Experiment Summary
Timeframe: 09/29/2020 - 10/05/2020
This is email 8 in the Constitution Day 2020 HFC for new LLS members. Does a mystery subject line that is vague on whether it’s part of the campaign perform better than a challenger that looks part of the campaign but includes personalization?
Research Question
Does a mystery subject line that is vague on whether it’s part of the campaign perform better than a challenger that looks part of the campaign but includes personalization?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | The fun isn't over... yet | 24.5% | ||
| T1: | Mr. Frontier, here's where we stand | 26.2% | 7.1% | 100.0% |
This experiment has a required sample size of 4,803 in order to be valid. Since the experiment had a total sample size of 632,786, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
7.1% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
Personalization is so important it trumps compensating for perceived email campaign fatigue.