How testing a new messaging approach affects conversion rate
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Timeframe: 10/23/2015 - 10/30/2015
As part of their membership campaign, The Heritage Foundation wanted to determine which messaging approach would be most appealing to their donors. At the start of the campaign, we created three landing page treatments based upon two new hypotheses of what the members would respond best to.
Our current control version focused on the current situation in American politics and framed it around the negative aspects of the “establishment.” Alternatively, we wanted to test a more positive tone using the same framework. The third treatment was a far more radical redesign using a more modern design that pointed to historical accomplishments and touted the benefits of membership.
Research Question
Which messaging approach will be most effective at converting visitors to members and increasing revenue?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Negative Focus on the "Establishment" | 2.1% | ||
| T1: | Positive Messaging | 3.4% | 60.5% | 98.3% |
| T2: | Past Accomplishments and Benefits | 4.0% | 87.2% | 99.9% |
This experiment has a required sample size of 925 in order to be valid. Since the experiment had a total sample size of 6,063, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 87.2% increase in conversion rate
× 11.9% increase in average gift
109.5% increase in revenue
Key Learnings
Both variants resulted in a statistically significant lift with the more radical redesign achieving a 87.2% lift over the control. This would lead us to hypothesize that the constituency is more likely to respond to a positive messaging approach. They may have frustrations with the current state of American politics but this is not what motivates them to give. They are responding to a message of hope and positive change.