How a text-based app promotion popup performed against an image-based popup
CaringBridge
CaringBridge offers free personal, protected websites for people to easily share updates and receive support and encouragement from their community during a health journey. Every 7 minutes, a CaringBridge website is created for someone experiencing a health event.
Experiment Summary
Ended On: 04/21/2020
CaringBridge was running popups to promote their app to users who had visited six or more times. Their control popup (on mobile devices only), used the official App Store logos for iPhone and Android, and only contained a single sentence of value proposition, which was a command: Download the mobile app now. They wanted to see if a popup with more visual contrast that engaged the visitor by asking a question would increase clickthrough rate. Additionally, they added a line of value proposition language to show what downloading the app would DO—allow them to “never miss an update your loved ones”. Because there was reduced space, they made a smaller button that prioritized iOS and added a text link below to for Android.
Research Question
Will a text-based app promotion popup increase clickthrough rate against an image-based popup?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.34% | ||
| T1: | Treatment #1 | 2.0% | 491.1% | 100.0% |
This experiment has a required sample size of 458 in order to be valid. Since the experiment had a total sample size of 2,080,085, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
491.1% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The text-based popup with the value proposition language increased clickthrough rate by 491.1%. Although actual app download numbers were impossible to track through this test, the combination of increased visual contrast, added value proposition copy, and unevenly weighted buttons had a tremendous effect on the number of people who were interested in the app offer.