How a subject line containing content clarity influenced open rates
Hillsdale College
Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.
Experiment Summary
Timeframe: 10/24/2019 - 11/07/2019
For Hillsdale’s cultivation email that shared an anniversary video produced by the College, we wanted to test how a subject line containing clarity about the contents of the email would perform. The control version used a more personal subject line, but was vague and left some mystery surrounding the contents of the email. Our hypothesis was that increasing clarity about the contents of the email would increase the open rates.
Research Question
Will increasing clarity in the subject line increase open rates?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Grateful for your friendship | 24.9% | ||
| T1: | Brief anniversary video | 23.3% | -6.7% | 100.0% |
This experiment has a required sample size of 5,044 in order to be valid. Since the experiment had a total sample size of 269,369, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
6.7% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
While the more specific subject line email had a lower open rate, we actually saw an increase in the overall clickthrough rate of the email, demonstrating that the recipients who had been primed to receive a video were more likely to view and engage with the video than those who received the vague subject line.