How the tactic used in presenting the value proposition impacted The Daily Signal’s donor conversion rate
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Timeframe: 10/21/2016 - 12/21/2016
Through a series of previous experiments, we had discovered a value proposition for The Daily Signal that seemed to resonate with donor. This value proposition focused on the type of journalism provided by TDS on a daily basis. With this in mind, we wanted to see if we could find a better means to deliver this message.
We created two new treatment options:
- The first treatment took the form of a “Wikipedia-style” delivery that appeared in the middle of every article. It devoted more space to cohesively lay out the reasons to support The Daily Signal.
- The second treatment took the form of an inline-survey to readers. The idea was to get the readers to say whether they valued the content. If they did, they would then be shown the donation value proposition as the next step in the process. If they selected they did not like it, they would be shown a comment box to submit the reasons why.
With such a high level of traffic, we were able to launch these three options at the same time over a period of a month.
Research Question
Which tactic in presenting the value proposition will lead to the greatest increase in donor conversion?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Inline Value Proposition | 0.01% | ||
| T1: | Dear Reader... | 0.10% | 641.9% | 100.0% |
| T2: | Inline Survey | 0.03% | 91.7% | 92.5% |
This experiment has a required sample size of 7,075 in order to be valid. Since the experiment had a total sample size of 241,871, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 641.9% increase in conversion rate
× 215.1% increase in average gift
2,237.7% increase in revenue
Key Learnings
The experiment produced significant lifts in both click through rate and donation rate. The table above lists the individual conversion rates from reader to donor. The click through rates followed similar trends. This results in two key learnings:
- By increasing the process-level value of donating (fully laying out the reasons to give), both treatments were able to outperformed the control.
- The “Dear Reader” version had the highest impact on clickthrough rate (256% lift) and donor conversion rates (642% lift).