How a relevant subject line affected housefile engagement for a cultivation email
Texas Public Policy Foundation
Experiment Summary
Ended On: 04/05/2021
We were noticing a recent decline in open rate for a weekly cultivation email series to the housefile so we wanted to test some subject line elements to see how we could improve how the audience was responding to different subject lines and increase engagement with the cultivation emails. We established a control—“An interesting interview”—using the element of mystery to invoke curiosity and a treatment—“Are you familiar with Mattress Mack?”—using the word you to create relevancy for the email recipients. We set up the A/B split test to see which of the two elements had better performance so we can continue to adjust subject lines for future emails in the series by using what works for this particular audience.
Research Question
We believe that introducing “you” to add relevance in a subject line for a cultivation series to the housefile will achieve an increase in open rate.
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | An interesting interview | 12.9% | ||
| T1: | Are you familiar with Mattress Mack? | 12.1% | -6.3% | 61.6% |
This experiment has a required sample size of 12,793 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
The experiment was set up in Hubspot to monitor results for the first 4 hours after the email test was sent to 50% of the total audience and then send the email with the most opens to the remainder of the file. Because of this, we did not have a large enough sample size to validate the experiment. The treatment saw a 6.3% decrease in traffic, but we did not reach a statistically significant level of confidence to draw any conclusions based on these results. Moving forward, we will want to run a similar test as a split to the full housefile rather than just half of it so we can validate results and find out how to best optimize the subject lines for this audience.