How a supportive top banner affects clickthrough rate on an exit-intent desktop popup Experiment ID: #25326

CaringBridge

CaringBridge offers free personal, protected websites for people to easily share updates and receive support and encouragement from their community during a health journey. Every 7 minutes, a CaringBridge website is created for someone experiencing a health event.

Experiment Summary

Ended On: 8/15/2020

CaringBridge had been running a popup that showed up on the 8th visit and offered the reader the chance to become a CaringBridge partner, or recurring donor. They had seen other tests where adding in a single-line motivational top banner to the popup might increase motivation to click. They decided to test this across both an exit intent desktop banner and a forced-popup mobile banner to see if they could increase clickthrough rate.

Research Question

How will a supportive top banner affect clickthrough rate on an exit-intent desktop popup?

Design

C: Control
T1: Treatment #1

Results

Treatment Name Click Rate Relative Difference Confidence
C: Control 0.76%
T1: Treatment #1 0.48% -37.4% 100.0%

This experiment has a required sample size of 6,007 in order to be valid. Since the experiment had a total sample size of 52,285, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    37.4% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The data they got was interesting—and somewhat confusing. Adding the banner to the desktop exit intent popup reduced clickthrough rate by 37%. However, adding it to the forced mobile popup increased clickthrough rate by 22%. Both results had a corresponding effect on recurring donations.

This prompted two follow-up tests:

  1. Should CaringBridge be testing a forced popup on the desktop site? Perhaps the “exit intent” strategy lowers motivation enough that this added clarity encouraged more people to leave.
  2. Should CaringBridge actually move the banner to the monthly donation page to increase conversion on that page?

They launched each respective winner on the respective platform and set out to further test the inclusion of this messaging.


Experiment Documented by...

Jeff Giddens

Jeff is the President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.