Hoover Institution

How name familiarity in an email subject line affects open rate

Experiment ID: #17811

Hoover Institution

Experiment Summary

Timeframe: 08/01/2019 - 08/15/2019

Hoover Institution created a new content offer using an essay from famed economist Milton Friedman. They had created an intriguing subject line that asked a question that prompted the recipient to open the email. However, they wondered if including Friedman’s name at the beginning—though grammatically awkward—would increase recognition and get more people to open. They launched a quick test to determine a winner to roll out to the whole file.

Research Question

How will name familiarity in an email subject line affect open rate?

Design

C: What does “created equal” mean?
T1: Milton Friedman: What does “created equal” mean?

Results

  Treatment Name Open Rate Relative Difference Confidence
C: What does “created equal” mean? 20.2%
T1: Milton Friedman: What does “created equal” mean? 21.5% 6.7% 100.0%

This experiment has a required sample size of 6,926 in order to be valid. Since the experiment had a total sample size of 51,100, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    6.7% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The subject line with Milton Friedman increased open rate by 6.7%—which showed that familiarity (even if the sender is not familiar) in the subject line can be a benefit to get more people to see the offer. Future sends with Friedman will use this similar tactic, though there might be diminishing returns without future testing.


Experiment Documented by Allison Jones
Allison Jones is Senior Optimization Associate at NextAfter.

Question about experiment #17811

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.