How text on a call-to-action button affects engagement and clickthrough rate.
Heritage Action for America
Experiment Summary
Timeframe: 08/29/2014 - 11/11/2014
Heritage Action For America launched a name acquisition campaign through an online mini-game in 2012 called “How Conservative Are You?”. The game let people answer 10 questions about legislation, then compared them to their Congressman. The game also had a social element too — if shared on Facebook, the user’s friends could take the quiz and then compare scores. Heritage Action wanted to promote social sharing because of the multiplier effect it had on name acquisition.
We launched a test on the button that asked the user to share. The control asked them a question — “How conservative are your friends?” While that might pique the interest of the user, it’s unclear what the button does. We launched a treatment that teased the outcome of the social share — “Show me how my friends score”. We also clarified copy around the button to indicate what was happening.
Research Question
Does clarifying the intent of the social share button by indicating the outcome increase social sharing (and in doing so, increase name acquisition)?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | How conservative are your friends | 26.3% | ||
| T1: | Show me how my friends score | 82.7% | 214.7% | 100.0% |
This experiment has a required sample size of 4 in order to be valid. Since the experiment had a total sample size of 20,411, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
214.7% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The treatment delivered a 214.7% lift in social sharing — which meant that not only did more people share it, more of their friends saw it. This brought a tremendous lift in traffic to the site, and more names were acquired as a result. It served as a great learning and reminder that at the process level of value proposition, using “outcome” language to earn the click can increase overall CTR.