How indicating the time commitment of an email impacts open rate

Care Net

Experiment Summary

Timeframe: 10/27/2018 - 11/10/2018

For the weekly cultivation email sent out by Care Net, we wanted to see if we could increase the email engagement by letting subscribers know the time commitment required to consume the topic. This is something often seen on blog posts but we wanted to see if it would have a positive impact in email. To do this, we added ” (4 min read)” to the end of the subject line.

Research Question

Will adding the time commitment to the subject line increase open rate?

Results

This experiment has a required sample size of 197,556,259 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

We saw a nearly identical open rate between the two versions. After further reflection, it is our hypothesis that the longer subject line may have resulted in the “(4 min read)” not being shown in the inbox. Having this aspect cut off would have resulted in the two variants being identical.