How adding a journal entry step into the new site signup process affects first journal creation rate

CaringBridge

CaringBridge offers free personal, protected websites for people to easily share updates and receive support and encouragement from their community during a health journey. Every 7 minutes, a CaringBridge website is created for someone experiencing a health event.

Experiment Summary

Ended On: 11/18/2022

CaringBridge was seeking to get more people to start sites, which keep them connected to loved ones through difficult health journeys. However, they did not just want people to start sites, they wanted them to write their first journal, which constitutes a “used site” and leads to more downstream engagement.

They decided to test placing a “write your first journal” prompt into the process of signing up for a new site, with an optional “skip this step” to get more people to write a journal.

They only tested this with 10% of the traffic to mitigate risk.

Research Question

We believe that adding a journal entry step for people registering for a new site will achieve an increase in used sites.

Design

C: Control

T1: Treatment #1

Results

This experiment has a required sample size of 42 in order to be valid. Since the experiment had a total sample size of 4,942, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 70.3% increase in conversion rate
× 0% increase in average gift

Key Learnings

The treatment, despite the small sample size, delivered a valid 70.3% increase in “used sites”, or sites with a first journal entry. Previously, the “write your first journal” step had been at the very end of the process, and far fewer people had actually taken the final step.

Interestingly, adding this “friction” into the process also