CaringBridge

How sticky bars affect conversion in non-match periods

Experiment ID: #17548

CaringBridge

CaringBridge offers free personal, protected websites for people to easily share updates and receive support and encouragement from their community during a health journey. Every 7 minutes, a CaringBridge website is created for someone experiencing a health event.

Experiment Summary

Timeframe: 06/16/2019 - 07/19/2019

CaringBridge had run sticky bars on their donation pages to promote matching grants during their campaign months. While this tactic had generally lifted response by reinforcing the match, they wanted to see if it could be effective in non-match times as well. They created two different treatments for the sticky bar—one that reminded them that CaringBridge is donor-supported, and one that thanked them for their generosity and included a photo and name of a CaringBridge staff member, for increased personalization. 

Research Question

How will sticky bars affect donor conversion in non-match periods?

Design

C: Control
T1: with header (personal)
T2: with header (KE Message)

Results

 Treatment NameConv. RateRelative DifferenceConfidenceAverage Gift
C: Control 2.3%
T1: with header (personal) 2.4%4.4% 70.4%
T2: with header (KE Message) 2.6%13.1% 99.7%

This experiment has a required sample size of 30,468 in order to be valid. Since the experiment had a total sample size of 143,839, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 13.1% increase in conversion rate
× 0% increase in average gift

Key Learnings

The experiment ran for a little more than a month, with little happening in the way of increased conversion. The team was about to shut it off, but they noticed that it was nearing a valid lift in conversion on mobile devices. After another ten days, it had achieved validity. 

The second treatment, with the thank-you messaging, increased donor conversion by 13.1%—but only on mobile devices. Because the sticky bar has much more visibility at the top of a smaller screen, it is much more impactful to that view. In fact, the sticky bar was producing negative results on desktop (-5%) and tablet (-11%) devices, which were holding back the aggregate increases. As mobile constitutes most of the traffic, this was a big lift in conversion, and an important reminder that device traffic must be split apart to see the true impact of tests. CaringBridge was able to roll this out to mobile visitors only using Google Optimize’s personalization feature. 


Experiment Documented by Jeff Giddens
Jeff Giddens is President of NextAfter.

Question about experiment #17548

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.