Canadian Taxpayers Federation

How adding clarity to the headline for a Dear Reader impacted email acquisition

Experiment ID: #169907

Canadian Taxpayers Federation

A not-for-profit citizen's advocacy group dedicated to lower taxes, less waste and accountable government.

Experiment Summary

Timeframe: 02/15/2024 - 02/23/2024

When advertising a newsletter on newsroom pages for CTF, we wondered if “A Note for our Readers” was causing banner blindness and not clearly articulating the offer to the page visitors. We set up a treatment that removed the headline and replaced it instead with the subheader which was “Join now to get the Taxpayer newsletter” and split traffic 50/50 between the two versions.

Research Question

We believe that adding clarity in the headline for page visitors will achieve an increase in email acquisition because the new headline is easier and quicker to understand what is being offered to the reader.


C: Control
T1: Variation 1


 Treatment NameConv. RateRelative DifferenceConfidenceAverage Gift
C: Control 0.49%
T1: Variation 1 1.1%126.5% 99.2%$13.33

This experiment has a required sample size of 1,587 in order to be valid. Since the experiment had a total sample size of 5,791, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 126.5% increase in conversion rate
× 0% increase in average gift

Key Learnings

By adding clarity in the headline for page visitors, we were able to significantly impact email acquisition rates as we observed a 126% increase in conversion with 99% level of confidence. Replacing a vague headline with a clear and specific message about the offer can lead to a substantial increase in conversions. We believe this increase happened because the new headline “Join now to get the Taxpayer newsletter” effectively communicated the value proposition to the readers, leading to a higher engagement rate. For future experiments, we will look to utilize clear and specific headlines that quickly convey the offer to potential subscribers. We will continue to test different variations of headlines to optimize for the highest email acquisition rates. We will also consider applying similar strategies to other marketing materials to enhance communication effectiveness with the target audience. As we think of future test ideas, we will continue to monitor and analyze the impact of headline changes on key metrics to further refine strategies for future campaigns.

Experiment Documented by Rebekah Josefy
Rebekah Josefy is an Optimization Director at NextAfter.

Question about experiment #169907

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.