CaringBridge

How sticky bars affect conversion in non-match periods

Experiment ID: #21354

CaringBridge

CaringBridge offers free personal, protected websites for people to easily share updates and receive support and encouragement from their community during a health journey. Every 7 minutes, a CaringBridge website is created for someone experiencing a health event.

Experiment Summary

Timeframe: 6/16/2019 - 1/1/1970

CaringBridge had run sticky bars on their donation pages to promote matching grants during their campaign months. While this tactic had generally lifted response by reinforcing the match, they wanted to see if it could be effective in non-match times as well. They created two different treatments for the sticky bar—one that reminded them that CaringBridge is donor-supported, and one that thanked them for their generosity and included a photo and name of a CaringBridge staff member, for increased personalization.

Research Question

How will sticky bars affect donor conversion in non-match periods?

Design

C: Control
T1: with header (personal),with header (KE Message)

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: 2.3%
T1: with header (personal),with header (KE Message) 0.24% -89.6% 100.0%

This experiment has a required sample size of 44 in order to be valid. Since the experiment had a total sample size of 4,862,495,215, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 89.6% decrease in conversion rate
× 0% increase in average gift

Key Learnings

The experiment ran for a little more than a month, with little happening in the way of increased conversion. The team was about to shut it off, but they noticed that it was nearing a valid lift in conversion on mobile devices. After another ten days, it had achieved validity.

The second treatment, with the thank-you messaging, increased donor conversion by 13.1%—but only on mobile devices. Because the sticky bar has much more visibility at the top of a smaller screen, it is much more impactful to that view. In fact, the sticky bar was producing negative results on desktop (-5%) and tablet (-11%) devices, which were holding back the aggregate increases. As mobile constitutes most of the traffic, this was a big lift in conversion, and an important reminder that device traffic must be split apart to see the true impact of tests. CaringBridge was able to roll this out to mobile visitors only using Google Optimize’s personalization feature.


Experiment Documented by NextAfter

Question about experiment #6317

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.