How a problem solution framework affects email clickthrough rate
Hillsdale College
Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.
Experiment Summary
Ended On: 08/14/2020
Hillsdale College launched a new course called introduction to Western philosophy. They had written and email prospecting treatment that describe to the course, and told the reader that they believed that philosophy was for everyone. The treatment email gave a preview of the course content and invited them to pre-register. However they wondered if framing the necessity for the course by describing the problem that our country finds itself in would increase the desire of the reader to take the course. The draft of the treatment that talked about how people have forgotten to reason and deliberate, instead choosing just to react. Then, the email positioned the course as a solution to this problem, as it would help build a defense against dangerous ideologies and prevent their spread. They launched an A/B test with three different email list brokers to determine a winner.
Research Question
How will a problem solution framework affect email clickthrough rate?
Design
Results
Treatment Name | Click Rate | Relative Difference | Confidence | |
---|---|---|---|---|
C: | Control | 1.0% | ||
T1: | Treatment #1 | 1.4% | 38.3% | 100.0% |
This experiment has a required sample size of 6,258 in order to be valid. Since the experiment had a total sample size of 38,627, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
38.3% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The treatment, with the problem solution framework, produced a 38.3% increase in click through rate. This Illustrated the principle that simply describing a course offering, or any sort of offer, is not enough. Copy must motivate the reader to want to take action, even if they do not perceive themselves to be in the group affected by the problem.
Understanding the reason a course was created, in this case, helped incentivize people to sign up for it—even though they knew less about what was in the course. This shows that emotion can overpower simple facts.
Question about experiment #34802
If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.