How inviting an email recipient to participate affects email open rate Experiment ID: #8087

Hillsdale College

Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.

Experiment Summary

Timeframe: 12/6/2017 - 12/13/2017

Hillsdale College recently began their calendar year end campaign. This year they decided to focus more specifically on their independence and how it enables them to fulfill their mission–without accepting EVEN ONE PENNY of government funding! One of their appeal emails focused on their presence in Washington D.C. through the Kirby Center. Before sending, they wondered: Will recipients be more motivated to open an email that invites them to participate in an action (in this case, helping teach policymakers about the constitution). They created an A/B subject line test and monitored the results.

Research Question

Will a subject line that invites the recipient to participate in an action increase email open rate?

MECLABS Conversion Factors Targeted

C = 4m + 3v + 2( i - f) - 2a ©

Copyright 2015, MECLABS

Design

C: Control
T1: Treatment 1

Results

Treatment Name Open Rate Relative Difference Confidence
C: Control 38.3%
T1: Treatment 1 36.3% -5.3% 100.0%

This experiment has a required sample size of 4,444 in order to be valid. Since the experiment had a total sample size of 38,790, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    5.3% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

Hillsdale College found that the treatment version actually decreased email open rate by about 5%. Though the relative impact of this test was small, it provides a valuable learning for future email subject line creation. Though the invitation to ‘Help teach policymakers the Constitution’ feels like a noble thing to ask recipients, it’s clear that it halted some in their inbox. It’s likely that the use of the word  ‘help’ struck some as a plea. This word makes people feel like the impetus is on them to do something. This can produce unnecessary anxiety in the mind of the recipient before they even have a chance to read the email.

This is an important thing to remember for any organization that is running a calendar year end campaign, and it will continue to inform Hillsdale College’s approach to email communication.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.