How promoting a petition in the “Dear Reader” call to action promotion on news articles impacted email signup rates
Americans for Prosperity
Experiment Summary
Timeframe: 01/12/2022 - 01/24/2022
As a part of experimenting with the promotion of newsroom articles, Americans for Prosperity wanted to test whether promoting a news-relevant petition in their “Dear Reader” promotion (which is a call to action at the end of a news article on their website) would be more productive than promoting their email subscription offer.
Research Question
We believe that promoting a relevant petition for article readers will achieve increase email signup rates.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Subscription Promotion | 6.8% | $0.00 | ||
| T1: | Relevant Petition | 20.0% | 196.1% | 100.0% | $0.00 |
This experiment has a required sample size of 51 in order to be valid. Since the experiment had a total sample size of 2,532, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 196.1% increase in conversion rate
× 0% increase in average gift
Key Learnings
With 100% level of confidence, we observed nearly a 3x increase in email signup rates (+196.1%) when promoting the news-relevant petition.
What’s also interesting to note is that the treatment petition offer also offers access to the subscription type offered in the control ad, but it’s the optional checkbox shown before the button to submit your petition form.
The data shows that even though this box is unchecked by default — meaning that if someone wanted to sign up for that subscription type, they would have to go through an extra step of checking the box before submitting the form — the treatment saw that 90.7% of the people who signed the treatment form also checked that box.
This means that the treatment with its unchecked, optional subscription offer, produced +58.8% more subscriptions to the primary offer made in the control experience (confidence level: 100%).
This means that we should look to promote petitions on the news articles moving forward. The next challenge will be to understand/determine how this offer performs when the news cycle shifts away from the offer in question.