Focus on the Family

How adding social proof and headline clarity affect email conversions

Experiment ID: #84651

Focus on the Family

Experiment Summary

Timeframe: 02/07/2022 - 04/10/2022

Focus on the Family has a number of free video series on topics around Faith, Marriage, and Parenting. The goal of putting out these free video series is to share free content around these topics and to collect names and email addresses of interested subscribers and then eventually nurture them towards a donation to the mission. The process is to promote the free video series on social media campaign, drive traffic to a landing page where they collect info like email address and name to send a link to access the free series, then upon a successful info collection, immediately send the subscriber to an instant donation page where they have the opportunity to provide a gift.

As we evaluate opportunities for optimization all along the marketing funnel, we look for metrics that show areas where we can make improvements in traffic conversion (CPMs, CTRs, CPCs), email conversions (Signup conversion rates), and instant donation conversions (ICRs). For this particular experiment, this email collection landing page is part of the overall campaign strategy and what we would refer to as the “middle of the marketing funnel”. The signup conversion rates on this campaign were lower than average, so we felt there was an opportunity to make improvements.

We decided to test if adding social proof under a landing page image and more clarity in the headline would result in a higher number of email signup conversions.

Research Question

We believe that adding social proof and headline clarity for readers of a landing page will achieve a higher number of conversions.

Design

C: Control

T1: Headline Test + Social Proof Callout

Results

	Treatment Name	Conv. Rate	Relative Difference	Confidence	Average Gift
C:	Control	7.2%			$0.00
T1:	Headline Test + Social Proof Callout	7.4%	2.1%	20.4%	$0.00

This experiment has a required sample size of 228,297 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

This experiment ran for a short duration being promoted through a social media campaign to get traffic. This campaign (like all Focus on the Family campaigns) is evaluated on a weekly basis for performance against a lofty ROI equation that requires a 1.1 ROI in order to continue getting paid media allocation and thus traffic to potentially validate. While there was not enough sample size to validate the experiment, we did see a small lift of around 2% for the treatment with a 20% level of confidence.

Due to the condensed timeframe for these experiments from media allocation, we experimented with two variants (headline clarity and social proof). While we typically like to limit experiments to 1 variable to isolate learnings, we opted for two variants in the same experiment to give a better opportunity for lift. With a low Level of Confidence (LoC - around 20%) and a small sample size, we cannot say definitively if the increased headline clarity or the social proof were the cause of the lift. We can say that we did see an increase in conversions, and gained learnings that we can continue to apply to future landing pages.

For future experiments with longer run times, we will try to isolate variants into single treatments for better learning.

Experiment Documented by NextAfter

Question about experiment #84651

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.