How using a question in a subject line impacted open rate for an email appeal
Texas Public Policy Foundation
Experiment Summary
Ended On:
TPPF was re-sending a matching gift appeal from a new sender and wanted to test whether using a question in the subject line would increase open rate on the email. The original appeal with the matching gift details had been sent from the organization’s executive director two weeks prior, so we decided to send a follow-up message from the development team calling attention to the previous email. The control subject line “Deadline today” contained a sense of urgency and mystery while the treatment “Did you see Kevin’s email?” utilized asking a direct, relevant question and also came across using a more humanized and personal approach.
Research Question
Will using a conversational question in a subject line impact the open rate of an email?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Deadline today | 24.2% | ||
| T1: | Did you see Kevin's email? | 26.7% | 10.3% | 97.9% |
This experiment has a required sample size of 2,337 in order to be valid. Since the experiment had a total sample size of 6,476, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
10.3% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The subject line with the question had an increase in open rate leading to a 10% lift in traffic when compared to the control. The treatment also saw an increase from 1.41% to 1.5% in the overall clickthrough rate. While we know this approach cannot be used for each email the organization sends, since the experiment results are valid, we can use these findings to strategically highlight especially significant emails to the housefile in the future to gain attention in the inbox and increase open and clickthrough rates.