Americans for Prosperity

How personal tone in a subject line affects open rate

Experiment ID: #4116

Americans for Prosperity

Experiment Summary

Timeframe: 05/15/2016 - 05/20/2016

Americans for Prosperity had scheduled a send for an appeal offering membership to their exclusive Torchbearer club. A Torchbearer membership is a big ask — a monthly recurring donation — so we hypothesized that the email must be as high-touch as possible.

This experience for the reader began with the subject line. Since the sender was Tim Phillips, the CEO of AFP, we wanted the subject line to be very personal. The control subject line said “Bringing the greatest message in a generation”, which was relevant to the content, but didn’t seem like a subject line that he might actually write. So we developed a treatment that matched the radical redesign of the treatment email: “An invitation”.

We then tested the two emails in equal segments to determine a winner.

Research Question

Will a more personal subject line result in a higher open rate?

Design

C: Control
T1: Personal Subject Line

Results

  Treatment Name Open Rate Relative Difference Confidence
C: Control 10.2%
T1: Personal Subject Line 14.8% 44.7% 100.0%

This experiment has a required sample size of 404 in order to be valid. Since the experiment had a total sample size of 14,613, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    44.7% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The treatment subject line increased open rate by 44.7%. This indicates that creating a personal email experience begins with the subject line. If the reader doesn’t believe it’s coming from a person, they might be less likely to open the email. We also tested a radical redesign of the email and a completely new copy approach, which also produced a lift in clickthrough rate. This is a reminder of a key principle from the Fundraiser’s Creed: people give to people, not organizations or marketing machines.


Experiment Documented by Jeff Giddens
Jeff Giddens is President of NextAfter.

Question about experiment #4116

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.