How traffic not coming from email was impacted by a Membership ask in The Daily Signal’s “Dear Reader” ads Experiment ID: #16030

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Timeframe: 4/18/2019 - 5/25/2019

Recently, we experimented with The Daily Signal’s “Dear Reader” ads and determined that return visitors were far more likely to respond to a Daily Signal ask, but that the message about “allying with President Trump” through “Activating your Heritage Membership” was better suited for first time visitors to the donation microsite.

In preparation of iterative testing against this hypothesis, we split traffic between “email-driven” and “non-email-driven” visitors through the “Dear Reader” ads, based upon the assumption that “email-driven” traffic was more than likely a “return” visitor to the donation microsite.

With the traffic split, and prior to installing an experiment to test The Daily Signal “Core” message, we wanted to make sure we had the best “Heritage ask” available since the control variant had been running for a number of months now to all audiences.

Research Question

Which “Dear Reader” ad would generate the best results for Heritage membership activation appeals?


C: Trump Ask (Activate Membership)
T1: Membership


Treatment Name Conv. Rate Relative Difference Confidence
C: Trump Ask (Activate Membership) 0.08%
T1: Membership 0.15% 83.7% 100.0%

This experiment has a required sample size of 19,434 in order to be valid. Since the experiment had a total sample size of 740,488, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 83.7% increase in conversion rate
× 0% increase in average gift

Key Learnings

With 100% level of confidence, we successfully validated that the “membership” treatment increased “Dear Reader” ad clicks by +83.7% over the control message for visitors not driven by email through the ads through to the donation page.

The purpose of the experiment was to see how this ad performed from a donor conversion and revenue standpoint, so below are the directionally accurate (albeit not validated) findings in those two metrics:

  • Donor Conversion Trend: +22.5% (LoC: 61.2%)
  • Revenue Trend: +30.1% (LoC: 86.8%)

Based upon these directionally accurate results and because this experiment has run for so long with a steady/similar trend, we decided to declare the “membership” treatment as the winner to now run up against the “TDS Core Ask” message.

Experiment Documented by...

Greg Colunga

Greg is Executive Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.