How adding urgency and relevancy to a subject line on an appeal impacted open rates
Texas Public Policy Foundation
Experiment Summary
Timeframe: 02/08/2021 - 02/22/2021
When sending a monthly appeal, we wanted to test if adding urgency and relevancy in the subject line would influence open rates. The control subject line “This legislative session” contained an element of mystery but we developed a treatment subject line “A critical year for the future of freedom for Texans” to draw attention to the timeliness and to make a personal connection to the reader, especially if they are a Texan concerned about freedom.
Research Question
We believe that adding relevancy and urgency for housefile recipients of a digital appeal will achieve an increase in open rate.
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | This legislative session | 36.7% | ||
| T1: | A critical year for the future of freedom for Texans | 23.4% | -36.3% | 100.0% |
This experiment has a required sample size of 91 in order to be valid. Since the experiment had a total sample size of 7,152, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
36.3% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
What we found was the subject line with additional detail reduced opens by 36.3% while the succinct, mysterious subject line outperformed the treatment with 100% level of confidence. One hypothesis is that “A critical year” indicates the recipient could take action at any time during the year versus “This legislative session” has a more finite timeframe thereby making it more relevant and timely. The element of mystery proved to drive open rates more than clarity in this specific instance. We’ll use this finding to develop additional subject line tests in the future.