How a subject line invoking curiosity impacted click rates on an email
American Cornerstone Institute
Experiment Summary
Timeframe: 10/27/2022 - 11/10/2022
When crafting an email to the housefile announcing the goal of 100,000 survey submissions had been reached, we wanted to test the subject line to get a learning about our housefile audience and what they respond to in order to help build future subject lines to use in an upcoming campaign. Would they be more motivated to open and take the intended desired action from an email with a subject line that did not include curiosity and rather gave clarity, revealing the contents of the message? Or would they be more compelled to open and click a message that related to an issue they care about and promises a reveal. In this case, the control was “100,000 … and counting” referring to the 100K survey submission goal and the treatment subject line was “This is great news”
Research Question
We believe that enhancing curiosity in a subject line for housefile email subscribers will achieve an increase in email engagement.
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | 100,000.... and counting | 1.6% | ||
| T1: | This is great news | 2.7% | 61.4% | 100.0% |
This experiment has a required sample size of 1,582 in order to be valid. Since the experiment had a total sample size of 81,631, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
61.4% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
We measured comparative open rates for each version of the email and saw a 4.3% lift in opens on the treatment version. However, we know that open rate is no longer a reliable metric due to inflated open rates being counted by audience members who are using Apple Mail. So we wanted to dig a little deeper to investigate the overall results on email engagement. We also measured the number of clicks on the Call to Action in the body copy for each version. Although the emails were identical aside from the subject line, we saw a 61.4% increase in click rate with 100% level of confidence on the treatment. We believe this is due to the positive momentum the subject line carries from “This is great news” into reminding people it wasn’t too late to submit their survey. If the reader was curious to see what the great news was, they had an increased chance of seeing the call to action inviting them to complete the survey as well. And since the message leaned into aligning with the subscriber’s values and highlighted social proof, they could now have a compelling reason to say “yes” to the ask and be a part of the continued great news.