How a text-based app promotion popup performed against an image-based popup Experiment ID: #20785

CaringBridge

CaringBridge offers free personal, protected websites for people to easily share updates and receive support and encouragement from their community during a health journey. Every 7 minutes, a CaringBridge website is created for someone experiencing a health event.

Experiment Summary

Ended On: 4/21/2020

CaringBridge was running popups to promote their app to users who had visited six or more times. Their control popup (on mobile devices only), used the official App Store logos for iPhone and Android, and only contained a single sentence of value proposition, which was a command: Download the mobile app now. They wanted to see if a popup with more visual contrast that engaged the visitor by asking a question would increase clickthrough rate. Additionally, they added a line of value proposition language to show what downloading the app would DO—allow them to “never miss an update your loved ones”. Because there was reduced space, they made a smaller button that prioritized iOS and added a text link below to for Android. 

Research Question

Will a text-based app promotion popup increase clickthrough rate against an image-based popup?

Design

C: Control
T1: Treatment #1

Results

Treatment Name Click Rate Relative Difference Confidence
C: Control 0.34%
T1: Treatment #1 2.0% 491.1% 100.0%

This experiment has a required sample size of 458 in order to be valid. Since the experiment had a total sample size of 2,080,085, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    491.1% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The text-based popup with the value proposition language increased clickthrough rate by 491.1%. Although actual app download numbers were impossible to track through this test, the combination of increased visual contrast, added value proposition copy, and unevenly weighted buttons had a tremendous effect on the number of people who were interested in the app offer. 


Experiment Documented by...

Jeff Giddens

Jeff is the President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.