How emphasizing the unknown impacts open rate Experiment ID: #21460
Timeframe: 12/19/2019 - 12/21/2019
EWTN was sending out an email as part of their calendar year end campaign. As with every email, we wanted to find a way to increase the engagement of the email audience. With this email, we decided to test two different subject lines. Both emphasized the “unknown” to try to increase intrigue but the control focused on the “work” and the treatment mentioned a “lesser known gift.”
We wanted to see which would increase opens and clicks.
Will a subject link that emphasizes a value-focused word (aka “gift”) increase engagement compared to an effort-focused word (aka “work”)?
|Treatment Name||Open Rate||Relative Difference||Confidence|
|C:||The work is never done||8.8%|
|T1:||A lesser known gift of the Holy Spirit||10.3%||17.2%||100.0%|
This experiment has a required sample size of 2,893 in order to be valid. Since the experiment had a total sample size of 194,621, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
17.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
After the email send was finished, we discovered that the value-focused headline that talked about the “lesser known gift” increased the number of opens by 17.2%. The increase in openers also led to a statistically significant 34% increase in clicks to the landing page.
This is a good reminder to focus your subject lines on the value being given to the reader instead of any allusion to work they may have to do when they open the email.