How emotive headline copy affects email acquisition rate Experiment ID: #6417

Hillsdale College

Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.

Experiment Summary

Timeframe: 3/10/2017 - 3/31/2017

Hillsdale College launched an updated version of their popular history course, American Heritage. They had received feedback on the original version of the course from many subscribers that the course “taught history the way it should be taught in school.” They decided to test whether this was a value proposition that resonated with others by crafting a similar headline: “Learn the history of America—the way it should be taught in school.” However, they feared that this might be too negative of a headline—and too focused on what is not taught in school, rather than what is taught in the course. So they created a treatment with highly emotive language: “Deepen your understanding of the incredible story of American Liberty.”

They kept all the other elements on the page the same, and launched an A/B headline test.

Research Question

Will emotive headline copy produce an increase in email acquisition rate?

MECLABS Conversion Factors Targeted

C = 4m + 3v + 2( i - f) - 2a ©

Copyright 2015, MECLABS

Design

C: Control
T1: Treatment 1

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Control 13.1%
T1: Treatment 1 35.3% 168.7% 100.0%

This experiment has a required sample size of 28 in order to be valid. Since the experiment had a total sample size of 2,356, and the level of confidence is above 95%, the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 168.7% increase in conversion rate
× 0% increase in average gift

Key Learnings

The treatment headline with the emotive language produced a 168.7% lift over the headline related to what is not taught in schools. This produced a valuable learning about motivation to enroll in the free course: we must orient our language around what the course does for the user, rather than align with a belief (even if it is true) about what the course replaces in the public sphere.

When a visitor comes to the page from an ad, they are motivated to do something, and our language must continue that momentum rather than causing their mind to drift elsewhere. Even if they align with the value proposition, a headline that doesn’t focus on what the course will do for them can mentally distract them from the ultimate goal of enrollment.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Jeff Giddens

Jeff is a Senior Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.