How a softer call to action affects conversion
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Timeframe: 03/14/2016 - 03/18/2016
The Heritage Foundation was promoting their Membership Card to non-members giving them the opportunity to become a member.
Anticipating that the direct ask to “become a member” would receive fewer (but more motivated) clicks from the email, they wanted to test whether the added step and call-to-action to “confirm the name on your card” would generate more clicks, and result in more donations.
Since they weren’t making a direct ask, they created a different offer flow for this treatment. They created a two-step form for the treatment that started with a simple “confirm your name” step (pre-populated with the visitor’s name), and then took them to complete their donation. Because the visitor was indicating their intent to become a member by completing the first step, they removed much of the sales copy on the donation page in order to remove friction from the process.
Research Question
Will a softer call to action generate more donations?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Direct Ask | 4.5% | ||
| T1: | Confirm your name | 2.2% | -51.6% | 100.0% |
This experiment has a required sample size of 422 in order to be valid. Since the experiment had a total sample size of 5,856, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 51.6% decrease in conversion rate
× 0% increase in average gift
Key Learnings
The softer call to action generated a 54% increase in clicks, but a 51.6% decrease in donations. This revealed a key insight: all traffic is not the same. Visitors come with different motivations, and the traffic to the treatment flow was not at all “sold”.
This prompted a second test to see if reintroducing the value proposition copy on the follow-on donation form would affect the results.