How a more casual subject line affects click through rate | NextAfter
Texas State Historical Association

How a more casual subject line affects click through rate

Experiment ID: #2932

Texas State Historical Association

Experiment Summary

Ended On: 12/31/2015

During their calendar year end fundraising campaign, the Texas State Historical Association sent a series of emails right up until the final day of the year. In this final email, we wanted to highlight the urgency of the appeal.  Our standard approach would be to use the subject line that reminded people of the limited time available.  We had a hypothesis that this language would appear gimmicky and may turn off potential donors. As a result, we decided to test a separate subject line that would appear more genuine and have a personal tone.

Research Question

Which subject lines will have the greatest impact on open and click through rates?


C: Are you with us?
T1: It is important that we remember...


Treatment Name Click Rate Relative Difference Confidence
C: Are you with us? 0.32%
T1: It is important that we remember... 0.50% 58.6% 95.1%

This experiment has a required sample size of 9,054 in order to be valid. Since the experiment had a total sample size of 18,247, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    58.6% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

While it removed the urgency that we strive to create with year end campaigns, the “It is important that we remember…” subject line conveyed a more personal tone. This tone was able to generate a 5.5% increase to the open rate. Not only that, but the constituency that opened were more willing to consume the message of the email which resulted in a 58.6% increase in visitors to the website.

Experiment Documented by NextAfter

Question about experiment #2932

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.