Hillsdale College

How a problem solution framework affects email clickthrough rate

Experiment ID: #34802

Hillsdale College

Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.

Experiment Summary

Ended On: 08/14/2020

Hillsdale College launched a new course called introduction to Western philosophy. They had written and email prospecting treatment that describe to the course, and told the reader that they believed that philosophy was for everyone. The treatment email gave a preview of the course content and invited them to pre-register. However they wondered if framing the necessity for the course by describing the problem that our country finds itself in would increase the desire of the reader to take the course. The draft of the treatment that talked about how people have forgotten to reason and deliberate, instead choosing just to react. Then, the email positioned the course as a solution to this problem, as it would help build a defense against dangerous ideologies and prevent their spread. They launched an A/B test with three different email list brokers to determine a winner.

Research Question

How will a problem solution framework affect email clickthrough rate?

Design

C: Control
T1: Treatment #1

Results

 Treatment NameClick RateRelative DifferenceConfidence
C: Control 1.0%
T1: Treatment #1 1.4%38.3% 100.0%

This experiment has a required sample size of 6,258 in order to be valid. Since the experiment had a total sample size of 38,627, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    38.3% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The treatment, with the problem solution framework, produced a 38.3% increase in click through rate. This Illustrated the principle that simply describing a course offering, or any sort of offer, is not enough. Copy must motivate the reader to want to take action, even if they do not perceive themselves to be in the group affected by the problem.

Understanding the reason a course was created, in this case, helped incentivize people to sign up for it—even though they knew less about what was in the course. This shows that emotion can overpower simple facts. 


Experiment Documented by Jeff Giddens
Jeff Giddens is President of NextAfter.

Question about experiment #34802

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.