How personalization in the subject line impacts engagement

Hillsdale College

Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.

Experiment Summary

Ended On: 02/07/2019

Hillsdale College recently sent out an email that offered the opportunity for the recipient to receive a pocket-sized Constitution booklet for absolutely no cost. The initial email was opened and clicked through by many, but a follow-up email was necessary to ensure as many people as possible had the chance to get their free pocket Constitution. Two distinct subject lines were crafted for this follow-up email, and the results regarding the effectiveness of these subject lines were recorded.

Research Question

Which subject line will achieve a higher open rate: one with personalization, a call to action, and a synonym for the word “free”? Or one without personalization, phrased as a question, and using the word “free”?

Results

This experiment has a required sample size of 1,316 in order to be valid. Since the experiment had a total sample size of 182,532, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    30.6% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

As predicted, the treatment subject line, was effective in achieving a higher open rate than the control subject line. In fact, the open rate for the treatment was 2.03% higher than the control. The treatment contained personalization, a synonym for the word “free,” and a call to action instead of a question. The combination of all of these resulted in a higher likelihood of the receiver opening the email.
The main purpose of the experiment was to measure the difference in open rates between the two subject lines, and those results were conclusive. However, the control subject line did achieve a higher click-through rate than the treatment. Looking into why this was the case could be the basis for future research.