How a more casual subject line affects click through rate Experiment ID: #2932
Texas State Historical Association
Ended On: 12/31/2015
During their calendar year end fundraising campaign, the Texas State Historical Association sent a series of emails right up until the final day of the year. In this final email, we wanted to highlight the urgency of the appeal. Our standard approach would be to use the subject line that reminded people of the limited time available. We had a hypothesis that this language would appear gimmicky and may turn off potential donors. As a result, we decided to test a separate subject line that would appear more genuine and have a personal tone.
Which subject lines will have the greatest impact on open and click through rates?
|Treatment Name||Click Rate||Relative Difference||Confidence|
|C:||Are you with us?||0.32%|
|T1:||It is important that we remember...||0.50%||58.6%||95.1%|
This experiment has a required sample size of 9,054 in order to be valid. Since the experiment had a total sample size of 18,247, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
58.6% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
While it removed the urgency that we strive to create with year end campaigns, the “It is important that we remember…” subject line conveyed a more personal tone. This tone was able to generate a 5.5% increase to the open rate. Not only that, but the constituency that opened were more willing to consume the message of the email which resulted in a 58.6% increase in visitors to the website.