How adding urgency and relevancy to a subject line on an appeal impacted open rates
Texas Public Policy Foundation
Timeframe: 02/08/2021 - 02/22/2021
When sending a monthly appeal, we wanted to test if adding urgency and relevancy in the subject line would influence open rates. The control subject line “This legislative session” contained an element of mystery but we developed a treatment subject line “A critical year for the future of freedom for Texans” to draw attention to the timeliness and to make a personal connection to the reader, especially if they are a Texan concerned about freedom.
We believe that adding relevancy and urgency for housefile recipients of a digital appeal will achieve an increase in open rate.
|Treatment Name||Open Rate||Relative Difference||Confidence|
|C:||This legislative session||36.7%|
|T1:||A critical year for the future of freedom for...||23.4%||-36.3%||100.0%|
This experiment has a required sample size of 91 in order to be valid. Since the experiment had a total sample size of 7,152, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
36.3% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
What we found was the subject line with additional detail reduced opens by 36.3% while the succinct, mysterious subject line outperformed the treatment with 100% level of confidence. One hypothesis is that “A critical year” indicates the recipient could take action at any time during the year versus “This legislative session” has a more finite timeframe thereby making it more relevant and timely. The element of mystery proved to drive open rates more than clarity in this specific instance. We’ll use this finding to develop additional subject line tests in the future.
Question about experiment #48311
If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.