How personal tone in a subject line affects open rate Experiment ID: #4116

Americans for Prosperity

Experiment Summary

Timeframe: 5/15/2016 - 5/20/2016

Americans for Prosperity had scheduled a send for an appeal offering membership to their exclusive Torchbearer club. A Torchbearer membership is a big ask — a monthly recurring donation — so we hypothesized that the email must be as high-touch as possible.

This experience for the reader began with the subject line. Since the sender was Tim Phillips, the CEO of AFP, we wanted the subject line to be very personal. The control subject line said “Bringing the greatest message in a generation”, which was relevant to the content, but didn’t seem like a subject line that he might actually write. So we developed a treatment that matched the radical redesign of the treatment email: “An invitation”.

We then tested the two emails in equal segments to determine a winner.

Research Question

Will a more personal subject line result in a higher open rate?

Design

C: Control
T1: Personal Subject Line

Results

Treatment Name Open Rate Relative Difference Confidence
C: Control 10.2%
T1: Personal Subject Line 14.8% 44.7% 100.0%

This experiment has a required sample size of 404 in order to be valid. Since the experiment had a total sample size of 14,613, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    44.7% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The treatment subject line increased open rate by 44.7%. This indicates that creating a personal email experience begins with the subject line. If the reader doesn’t believe it’s coming from a person, they might be less likely to open the email. We also tested a radical redesign of the email and a completely new copy approach, which also produced a lift in clickthrough rate. This is a reminder of a key principle from the Fundraiser’s Creed: people give to people, not organizations or marketing machines.


Get the Detailed Case Study

Get a copy of the slides that highlight the different elements of the experiment, the variables that were changed, and the key takeaways and learnings that you can apply to your own organization.

Experiment Documented by...

Jeff Giddens

Jeff is the President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.