How emotive headline copy affects email acquisition rate Experiment ID: #6417
Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.
Timeframe: 3/10/2017 - 3/31/2017
Hillsdale College launched an updated version of their popular history course, American Heritage. They had received feedback on the original version of the course from many subscribers that the course “taught history the way it should be taught in school.” They decided to test whether this was a value proposition that resonated with others by crafting a similar headline: “Learn the history of America—the way it should be taught in school.” However, they feared that this might be too negative of a headline—and too focused on what is not taught in school, rather than what is taught in the course. So they created a treatment with highly emotive language: “Deepen your understanding of the incredible story of American Liberty.”
They kept all the other elements on the page the same, and launched an A/B headline test.
Will emotive headline copy produce an increase in email acquisition rate?
MECLABS Conversion Factors Targeted
C = 4m + 3v + 2( i - f) - 2a ©
Copyright 2015, MECLABS
|Treatment Name||Conv. Rate||Relative Difference||Confidence|
This experiment has a required sample size of 28 in order to be valid. Since the experiment had a total sample size of 2,356, and the level of confidence is above 95%, the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 168.7% increase in conversion rate
× 0% increase in average gift
The treatment headline with the emotive language produced a 168.7% lift over the headline related to what is not taught in schools. This produced a valuable learning about motivation to enroll in the free course: we must orient our language around what the course does for the user, rather than align with a belief (even if it is true) about what the course replaces in the public sphere.
When a visitor comes to the page from an ad, they are motivated to do something, and our language must continue that momentum rather than causing their mind to drift elsewhere. Even if they align with the value proposition, a headline that doesn’t focus on what the course will do for them can mentally distract them from the ultimate goal of enrollment.