How adding clarity to the headline for a Dear Reader impacted email acquisition
Canadian Taxpayers Federation
A not-for-profit citizen's advocacy group dedicated to lower taxes, less waste and accountable government.
Experiment Summary
Timeframe: 02/15/2024 - 02/23/2024
When advertising a newsletter on newsroom pages for CTF, we wondered if “A Note for our Readers” was causing banner blindness and not clearly articulating the offer to the page visitors. We set up a treatment that removed the headline and replaced it instead with the subheader which was “Join now to get the Taxpayer newsletter” and split traffic 50/50 between the two versions.
Research Question
We believe that adding clarity in the headline for page visitors will achieve an increase in email acquisition because the new headline is easier and quicker to understand what is being offered to the reader.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | 0.49% | |||
| T1: | Variation 1 | 1.1% | 126.5% | 99.2% | $13.33 |
This experiment has a required sample size of 1,587 in order to be valid. Since the experiment had a total sample size of 5,791, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 126.5% increase in conversion rate
× 0% increase in average gift
Key Learnings
By adding clarity in the headline for page visitors, we were able to significantly impact email acquisition rates as we observed a 126% increase in conversion with 99% level of confidence. Replacing a vague headline with a clear and specific message about the offer can lead to a substantial increase in conversions. We believe this increase happened because the new headline “Join now to get the Taxpayer newsletter” effectively communicated the value proposition to the readers, leading to a higher engagement rate. For future experiments, we will look to utilize clear and specific headlines that quickly convey the offer to potential subscribers. We will continue to test different variations of headlines to optimize for the highest email acquisition rates. We will also consider applying similar strategies to other marketing materials to enhance communication effectiveness with the target audience. As we think of future test ideas, we will continue to monitor and analyze the impact of headline changes on key metrics to further refine strategies for future campaigns.