Texas Public Policy Foundation

How adding urgency and relevancy to a subject line on an appeal impacted open rates

Experiment ID: #48311

Texas Public Policy Foundation

Experiment Summary

Timeframe: 02/08/2021 - 02/22/2021

When sending a monthly appeal, we wanted to test if adding urgency and relevancy in the subject line would influence open rates. The control subject line “This legislative session” contained an element of mystery but we developed a treatment subject line “A critical year for the future of freedom for Texans” to draw attention to the timeliness and to make a personal connection to the reader, especially if they are a Texan concerned about freedom.

Research Question

We believe that adding relevancy and urgency for housefile recipients of a digital appeal will achieve an increase in open rate.

Design

C: This legislative session
T1: A critical year for the future of freedom for Texans

Results

 Treatment NameOpen RateRelative DifferenceConfidence
C: This legislative session 36.7%
T1: A critical year for the future of freedom for Texans 23.4%-36.3% 100.0%

This experiment has a required sample size of 91 in order to be valid. Since the experiment had a total sample size of 7,152, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    36.3% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

What we found was the subject line with additional detail reduced opens by 36.3% while the succinct, mysterious subject line outperformed the treatment with 100% level of confidence. One hypothesis is that “A critical year” indicates the recipient could take action at any time during the year versus “This legislative session” has a more finite timeframe thereby making it more relevant and timely. The element of mystery proved to drive open rates more than clarity in this specific instance. We’ll use this finding to develop additional subject line tests in the future.


Experiment Documented by Rebekah Josefy
Rebekah Josefy is an Optimization Director at NextAfter.

Question about experiment #48311

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.