How clarifying the value proposition affects email acquisition rate
Hillsdale College
Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.
Experiment Summary
Timeframe: 11/14/2015 - 11/24/2015
Hillsdale College was promoting their popular Constitution 101 course with a page that had been through a series of tests centered around copy and the email acquisition form. Through all of these, the top banner had remained the same. They hypothesized that the separation of the headline from the rest of the copy might be creating a “break” between the two sections. They designed a treatment that removed the “banner” section and closely joined it to the body copy, in an effort to reduce friction and increase the clarity of the value proposition. Also, they tested an alternate headline that tested the value proposition offering “Instant Free Access” to the course, hoping both of these changes would result in an increase in emails acquired.
Research Question
Can we affect email acquisition rate by positioning the headline in greater proximity to the body copy and increasing the clarity of the value proposition?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control - Banner & Know the Constitution | 37.5% | ||
| T1: | Instant Free Access - No Banner | 40.0% | 6.6% | 92.5% |
| T2: | Know the constitution - No Banner | 41.0% | 9.2% | 98.7% |
This experiment has a required sample size of 2,327 in order to be valid. Since the experiment had a total sample size of 7,473, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 9.2% increase in conversion rate
× 0% increase in average gift
Key Learnings
Hillsdale saw a 9.2% lift in email acquisition from these simple design changes, showing that proximity matters to visitors. If we separate the attention-grabbing headline too far from the body copy, it reduces the likelihood that they visitors will read both. Additionally, there were friction factors at work as well — reducing the overall height of the page brought the signup form closer to the top of the page, which led to reduced scrolling without affecting the amount of value proposition copy on the page.