How a command instead of a question affects clickthrough rate Experiment ID: #21031

CaringBridge

CaringBridge offers free personal, protected websites for people to easily share updates and receive support and encouragement from their community during a health journey. Every 7 minutes, a CaringBridge website is created for someone experiencing a health event.

Experiment Summary

Ended On: 5/7/2020

CaringBridge had run experiments on a sticky bar promoting its mobile app. They had discovered that a colored, conversational popup outperformed a low-contrast logo-based design. However, they were concerned that the question might cause cognitive friction in the mind of the user as they attempted to answer it rather than respond to or consider the offer. They decided to test a command instead, to present the user with a suggested decision rather than an open-ended question.

Research Question

How will a command instead of a question affect clickthrough rate?

Design

C: Control
T1: Treatment #1

Results

Treatment Name Click Rate Relative Difference Confidence
C: Control 1.3%
T1: Treatment #1 1.7% 34.7% 98.8%

This experiment has a required sample size of 5,520 in order to be valid. Since the experiment had a total sample size of 20,229, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    34.7% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The treatment, with the command, produced a 34.7% lift in clickthrough rate. This appears to confirm that the question did cause some users to pause, or answer negatively, which might have impacted the decision to click. Additionally, moving the word “try” to the front of the sentence might have lowered the perceived anxiety or commitment to the offer as well.


Experiment Documented by...

Jeff Giddens

Jeff is the President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.