How personalizing an email subject line impacts opens
Live Action
Experiment Summary
Timeframe: 10/16/2017 - 10/17/2017
Live Action was running a high-urgency campaign around Twitter banning their pro-life message and ads from it’s platform. The first email in the campaign was a petition to their housefile asking people to add their name to the list of pro-life advocates. They did a resend of that same email and wanted to test the subject line of the resend to see if they could get more people to open the email, read the contents in it, and click-through to the landing page. The control used a similar approach that we had worked well in a previous campaign, “Did you see Lila’s email?” The treatment used a personalization tactic in the subject line, “[FIRSTNAME], your name is missing from our Twitter petition”.
Research Question
Which subject line would increase opens and potentially get more clicks?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Did you see Lila's email? | 19.1% | ||
| T1: | [FIRSTNAME], your name is missing... | 19.9% | 4.2% | 100.0% |
This experiment has a required sample size of 18,535 in order to be valid. Since the experiment had a total sample size of 140,121, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
4.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The treatment increased opens by 4.2%. While the control subject line was engaging, the treatment directly told called the end reader out and told them the purpose of the email. Even though this subject line experiment only slightly impacted open rate, it also impact click-rate. The treatment also helped increase the amount of people clicking through to the landing page because it reinforced the value proposition of the email content.