How a new testimonial impacted clicks on a sticky bar

Focus on the Family

Focus on the Family is a global Christian ministry dedicated to helping families thrive. We provide help and resources for couples to build healthy marriages that reflect God's design, and for parents to raise their children according to morals and values grounded in biblical principles.

Experiment Summary

Timeframe: 02/02/2021 - 03/06/2021

When reviewing surveys from people who had completed a 14-day trial, we found a testimonial that indicated the different skills their children built as a result of listening to Adventures in Odyssey episodes. We decided to include this testimonial as a form of social proof in the sticky bar to see if tangible outcomes in life skill areas such as “conflict resolution, compassion and addressing fears” would be motivating for parents considering beginning a free trial for their children. We split traffic between this treatment and the currently running testimonial that was implemented as the control after it had lifted results in a prior experiment.

Research Question

We believe that demonstrating tangible outcomes in the value proposition for parents considering a free trial will achieve an increase in free trial signups.

Results

This experiment has a required sample size of 947,600 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

We ran the experiment for a month, but we did not validate this test. While we saw varying results across multiple segments, only a few individual segments did validate. We’ll look further into the mobile vs. desktop experience for future tests since it looked like these segments had opposing effects. We can also look into adding variable copy for segments like paid search visitors on any additional treatments.

Observations of the impact on other visitor segments in the experiment:

• Direct Traffic visitors had a 41.2% increase in begin free trial signup with an 87% level of confidence.
• Paid Search visitors had a 20.5% decrease in begin free trial signup with a 97% level of confidence.
• Social visitors had a 2.7% increase in begin free trial signup with a 92% level of confidence.
• Desktop visitors had a 20.4% increase in begin free trial signup with a 96% level of confidence.
• Mobile visitors had a 14.3% decrease in begin free trial signup with a 94% level of confidence.
• (Other) visitors had a 107.0% increase in begin free trial signup with a 91% level of confidence.
• Direct visitors had a 36.8% increase in begin free trial signup with a 96% level of confidence.