How a command instead of a question affects clickthrough rate
CaringBridge
CaringBridge offers free personal, protected websites for people to easily share updates and receive support and encouragement from their community during a health journey. Every 7 minutes, a CaringBridge website is created for someone experiencing a health event.
Experiment Summary
Ended On: 05/07/2020
CaringBridge had run experiments on a sticky bar promoting its mobile app. They had discovered that a colored, conversational popup outperformed a low-contrast logo-based design. However, they were concerned that the question might cause cognitive friction in the mind of the user as they attempted to answer it rather than respond to or consider the offer. They decided to test a command instead, to present the user with a suggested decision rather than an open-ended question.
Research Question
How will a command instead of a question affect clickthrough rate?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 1.3% | ||
| T1: | Treatment #1 | 1.7% | 34.7% | 98.8% |
This experiment has a required sample size of 5,520 in order to be valid. Since the experiment had a total sample size of 20,229, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
34.7% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The treatment, with the command, produced a 34.7% lift in clickthrough rate. This appears to confirm that the question did cause some users to pause, or answer negatively, which might have impacted the decision to click. Additionally, moving the word “try” to the front of the sentence might have lowered the perceived anxiety or commitment to the offer as well.