How charged language performs in a subject line
Leadership Institute
Experiment Summary
Timeframe: 08/07/2017 - 08/21/2017
The Leadership Institute works with colleges to combat leftis bias on campus. They wanted to know if giving a sneak peak to the email by naming the college in the subject line would generate more clicks than a subject line that included charged language that riles up their donor base.
Research Question
Will a subject line that includes the name of a specific school generate more clicks than subject line with charged language?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Students won't attend school because of racist professor | 6.2% | ||
| T1: | Do you remember what happened at Trinity College? | 3.8% | -39.5% | 96.5% |
This experiment has a required sample size of 604 in order to be valid. Since the experiment had a total sample size of 1,386, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
39.5% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The subject line that included charged language reached a 6.2% click rate, in comparison to the 3.8% click rate achieved by the subject line that included a specific name of a college. We can deduct that the subject line with charged language is the winner. Both of the subject lines received the same amount of opens, so to conclude this experiment in full, a future experiment should be ran with the same type of versions, optimized for opens.