How traffic not coming from email was impacted by a Membership ask in The Daily Signal’s “Dear Reader” ads
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Timeframe: 04/18/2019 - 05/25/2019
Recently, we experimented with The Daily Signal’s “Dear Reader” ads and determined that return visitors were far more likely to respond to a Daily Signal ask, but that the message about “allying with President Trump” through “Activating your Heritage Membership” was better suited for first time visitors to the donation microsite.
In preparation of iterative testing against this hypothesis, we split traffic between “email-driven” and “non-email-driven” visitors through the “Dear Reader” ads, based upon the assumption that “email-driven” traffic was more than likely a “return” visitor to the donation microsite.
With the traffic split, and prior to installing an experiment to test The Daily Signal “Core” message, we wanted to make sure we had the best “Heritage ask” available since the control variant had been running for a number of months now to all audiences.
Research Question
Which “Dear Reader” ad would generate the best results for Heritage membership activation appeals?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Trump Ask (Activate Membership) | 0.08% | ||
| T1: | Membership | 0.15% | 83.7% | 100.0% |
This experiment has a required sample size of 19,434 in order to be valid. Since the experiment had a total sample size of 740,488, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 83.7% increase in conversion rate
× 0% increase in average gift
Key Learnings
With 100% level of confidence, we successfully validated that the “membership” treatment increased “Dear Reader” ad clicks by +83.7% over the control message for visitors not driven by email through the ads through to the donation page.
The purpose of the experiment was to see how this ad performed from a donor conversion and revenue standpoint, so below are the directionally accurate (albeit not validated) findings in those two metrics:
- Donor Conversion Trend: +22.5% (LoC: 61.2%)
- Revenue Trend: +30.1% (LoC: 86.8%)
Based upon these directionally accurate results and because this experiment has run for so long with a steady/similar trend, we decided to declare the “membership” treatment as the winner to now run up against the “TDS Core Ask” message.