Texas Public Policy Foundation

How using a question in a subject line impacted open rate for an email appeal

Experiment ID: #65614

Texas Public Policy Foundation

Experiment Summary

Ended On:

TPPF was re-sending a matching gift appeal from a new sender and wanted to test whether using a question in the subject line would increase open rate on the email. The original appeal with the matching gift details had been sent from the organization’s executive director two weeks prior, so we decided to send a follow-up message from the development team calling attention to the previous email. The control subject line “Deadline today” contained a sense of urgency and mystery while the treatment “Did you see Kevin’s email?” utilized asking a direct, relevant question and also came across using a more humanized and personal approach.

Research Question

Will using a conversational question in a subject line impact the open rate of an email?


C: Deadline today
T1: Did you see Kevin's email?


 Treatment NameOpen RateRelative DifferenceConfidence
C: Deadline today 24.2%
T1: Did you see Kevin's email? 26.7%10.3% 97.9%

This experiment has a required sample size of 2,337 in order to be valid. Since the experiment had a total sample size of 6,476, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    10.3% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The subject line with the question had an increase in open rate leading to a 10% lift in traffic when compared to the control. The treatment also saw an increase from 1.41% to 1.5% in the overall clickthrough rate. While we know this approach cannot be used for each email the organization sends, since the experiment results are valid, we can use these findings to strategically highlight especially significant emails to the housefile in the future to gain attention in the inbox and increase open and clickthrough rates.

Experiment Documented by Rebekah Josefy
Rebekah Josefy is an Optimization Director at NextAfter.

Question about experiment #65614

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.