How fear of loss affects clickthrough rate on an app download popup Experiment ID: #21186

CaringBridge

CaringBridge offers free personal, protected websites for people to easily share updates and receive support and encouragement from their community during a health journey. Every 7 minutes, a CaringBridge website is created for someone experiencing a health event.

Experiment Summary

Ended On: 5/26/2020

CaringBridge was using a popup to advertise their mobile app to people who visited their site 5 times. They had proven that a conversational tone and simple design improved clickthrough rate. However, they knew that visitors weren’t just sitting around waiting to download an app—they had to understand what the app would do for them. The control copy had copy about “not missing an update”, but it wasn’t the first sentence. They wanted to test whether leading with fear of loss—”don’t miss an update”—would beat a simple request to try the app.

Research Question

Will introduction of “fear of loss” language increase clickthrough rates?

Design

C: Control
T1: Treatment #1

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Control 1.5%
T1: Treatment #1 1.7% 11.8% 99.1%

This experiment has a required sample size of 36,484 in order to be valid. Since the experiment had a total sample size of 139,380, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 11.8% increase in conversion rate
× 0% increase in average gift

Key Learnings

Prioritizing the language regarding “don’t miss out” did appear to strike a chord—increasing clickthroughs by 11.7%. However, there was another variable with this test that was introduced—the treatment used “new and improved” language


Experiment Documented by...

Jeff Giddens

Jeff is the President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.