How inviting an email recipient to participate affects email open rate
Hillsdale College
Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.
Experiment Summary
Timeframe: 12/06/2017 - 12/13/2017
Hillsdale College recently began their calendar year end campaign. This year they decided to focus more specifically on their independence and how it enables them to fulfill their mission–without accepting EVEN ONE PENNY of government funding! One of their appeal emails focused on their presence in Washington D.C. through the Kirby Center. Before sending, they wondered: Will recipients be more motivated to open an email that invites them to participate in an action (in this case, helping teach policymakers about the constitution). They created an A/B subject line test and monitored the results.
Research Question
Will a subject line that invites the recipient to participate in an action increase email open rate?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 38.3% | ||
| T1: | Treatment 1 | 36.3% | -5.3% | 100.0% |
This experiment has a required sample size of 4,444 in order to be valid. Since the experiment had a total sample size of 38,790, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
5.3% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
Hillsdale College found that the treatment version actually decreased email open rate by about 5%. Though the relative impact of this test was small, it provides a valuable learning for future email subject line creation. Though the invitation to ‘Help teach policymakers the Constitution’ feels like a noble thing to ask recipients, it’s clear that it halted some in their inbox. It’s likely that the use of the word ‘help’ struck some as a plea. This word makes people feel like the impetus is on them to do something. This can produce unnecessary anxiety in the mind of the recipient before they even have a chance to read the email.
This is an important thing to remember for any organization that is running a calendar year end campaign, and it will continue to inform Hillsdale College’s approach to email communication.