How a celebrity signer affects open and click rates
Leadership Institute
Experiment Summary
Ended On: 07/12/2018
The Leadership Institute has access to many high profile people in the conservative movement. They wanted to see if mentioning these people would impact their online presence. They decided to test a subject line that states Mike Huckabee has involvement with LI vs a vague subject line about something that makes students freeze when they hear it.
Research Question
Will a subject line that mentions a conservative celebrity increase open rates?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Mike Huckabee hosts LI's own | 1.8% | ||
| T1: | Students freeze up when they hear this | 3.9% | 121.3% | 100.0% |
This experiment has a required sample size of 458 in order to be valid. Since the experiment had a total sample size of 23,261, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
121.3% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The subject line that included Mike Huckabee garnered an 18.3% open rate, while the vague subject line about students earned a 20% open rate. This is a lift of 9.5%. There was a 121.3% lift in clicks for the email that had a student focus in the subject line.
This experiment tells us that LI email recipients care more about what is being discussed on campus than about a conservative celebrity hosting an LI employee on their show. However, these results could change depending on the celebrity signer. In the future, the Leadership Institute should look into how the subject line and the copy work together to generate opens and clicks.