How a more casual subject line affects click through rate Experiment ID: #2932

Texas State Historical Association

Experiment Summary

Ended On: 12/31/2015

During their calendar year end fundraising campaign, the Texas State Historical Association sent a series of emails right up until the final day of the year. In this final email, we wanted to highlight the urgency of the appeal.  Our standard approach would be to use the subject line that reminded people of the limited time available.  We had a hypothesis that this language would appear gimmicky and may turn off potential donors. As a result, we decided to test a separate subject line that would appear more genuine and have a personal tone.

Research Question

Which subject lines will have the greatest impact on open and click through rates?

Design

C: Are you with us?
T1: It is important that we remember...

Results

Treatment Name Click Rate Relative Difference Confidence
C: Are you with us? 0.32%
T1: It is important that we remember... 0.50% 58.6% 95.1%

This experiment has a required sample size of 9,054 in order to be valid. Since the experiment had a total sample size of 18,247, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    58.6% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

While it removed the urgency that we strive to create with year end campaigns, the “It is important that we remember…” subject line conveyed a more personal tone. This tone was able to generate a 5.5% increase to the open rate. Not only that, but the constituency that opened were more willing to consume the message of the email which resulted in a 58.6% increase in visitors to the website.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.





Related Experiments

This experiment is part of a series of experiments aimed at improving overall results. Take a look at some of the other iterations:


Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.