How a supportive top banner affects clickthrough rate on a mobile popup
CaringBridge
CaringBridge offers free personal, protected websites for people to easily share updates and receive support and encouragement from their community during a health journey. Every 7 minutes, a CaringBridge website is created for someone experiencing a health event.
Experiment Summary
Ended On: 08/18/2020
CaringBridge had been running a popup that showed up on the 8th visit and offered the reader the chance to become a CaringBridge partner, or recurring donor. They had seen other tests where adding in a single-line motivational top banner to the popup might increase motivation to click. They decided to test this across both an exit intent desktop banner and a forced-popup mobile banner to see if they could increase clickthrough rate.
Research Question
How will a supportive top banner affect clickthrough rate on a mobile popup?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 1.6% | ||
| T1: | Treatment #1 | 2.0% | 22.5% | 99.9% |
This experiment has a required sample size of 10,136 in order to be valid. Since the experiment had a total sample size of 56,054, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
22.5% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The data they got was interesting—and somewhat confusing. Adding the banner to the desktop exit intent popup reduced clickthrough rate by 37%. However, adding it to the forced mobile popup increased clickthrough rate by 22%. Both results had a corresponding effect on recurring donations.
This prompted two follow-up tests:
- Should CaringBridge be testing a forced popup on the desktop site? Perhaps the “exit intent” strategy lowers motivation enough that this added clarity encouraged more people to leave.
- Should CaringBridge actually move the banner to the monthly donation page to increase conversion on that page?
They launched each respective winner on the respective platform and set out to further test the inclusion of this messaging.