How emphasizing the unknown impacts open rate Experiment ID: #21460

EWTN

Experiment Summary

Timeframe: 12/19/2019 - 12/21/2019

EWTN was sending out an email as part of their calendar year end campaign. As with every email, we wanted to find a way to increase the engagement of the email audience. With this email, we decided to test two different subject lines. Both emphasized the “unknown” to try to increase intrigue but the control focused on the “work” and the treatment mentioned a “lesser known gift.”

We wanted to see which would increase opens and clicks.

Research Question

Will a subject link that emphasizes a value-focused word (aka “gift”) increase engagement compared to an effort-focused word (aka “work”)?

Design

C: The work is never done
T1: A lesser known gift of the Holy Spirit

Results

Treatment Name Open Rate Relative Difference Confidence
C: The work is never done 8.8%
T1: A lesser known gift of the Holy Spirit 10.3% 17.2% 100.0%

This experiment has a required sample size of 2,893 in order to be valid. Since the experiment had a total sample size of 194,621, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    17.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

After the email send was finished, we discovered that the value-focused headline that talked about the “lesser known gift” increased the number of opens by 17.2%. The increase in openers also led to a statistically significant 34% increase in clicks to the landing page.

This is a good reminder to focus your subject lines on the value being given to the reader instead of any allusion to work they may have to do when they open the email.


Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.