How sticky bars affect conversion in non-match periods
CaringBridge
CaringBridge offers free personal, protected websites for people to easily share updates and receive support and encouragement from their community during a health journey. Every 7 minutes, a CaringBridge website is created for someone experiencing a health event.
Experiment Summary
Timeframe: 06/16/2019 - 07/19/2019
CaringBridge had run sticky bars on their donation pages to promote matching grants during their campaign months. While this tactic had generally lifted response by reinforcing the match, they wanted to see if it could be effective in non-match times as well. They created two different treatments for the sticky bar—one that reminded them that CaringBridge is donor-supported, and one that thanked them for their generosity and included a photo and name of a CaringBridge staff member, for increased personalization.
Research Question
How will sticky bars affect donor conversion in non-match periods?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | 2.3% | $0.00 | ||
| T1: | with header (personal) | 2.4% | 4.4% | 70.4% | $0.00 |
| T2: | with header (KE Message) | 2.6% | 13.1% | 99.7% | $0.00 |
This experiment has a required sample size of 30,468 in order to be valid. Since the experiment had a total sample size of 143,839, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 13.1% increase in conversion rate
× 0% increase in average gift
Key Learnings
The experiment ran for a little more than a month, with little happening in the way of increased conversion. The team was about to shut it off, but they noticed that it was nearing a valid lift in conversion on mobile devices. After another ten days, it had achieved validity.
The second treatment, with the thank-you messaging, increased donor conversion by 13.1%—but only on mobile devices. Because the sticky bar has much more visibility at the top of a smaller screen, it is much more impactful to that view. In fact, the sticky bar was producing negative results on desktop (-5%) and tablet (-11%) devices, which were holding back the aggregate increases. As mobile constitutes most of the traffic, this was a big lift in conversion, and an important reminder that device traffic must be split apart to see the true impact of tests. CaringBridge was able to roll this out to mobile visitors only using Google Optimize’s personalization feature.