e3 Partners

How improving clarity of message affects email acquisition for returning visitors

Experiment ID: #87643

e3 Partners

Experiment Summary

Timeframe: 03/07/2022 - 04/10/2022

I Am Second has a goal of getting visitors to join their free community portal to gain access to more content, film archives, and access to life coaching. The experiment was set up to show a pop-up to returning visitors of their popular film pages, with the goal of getting them to join the community for greater engagement.

I Am Second wanted to know if adding clearer copy (w/ greater value articulation) to a pop-up shown to returning visitors would result in greater sign-ups to a community “portal”…even if it meant more text.

The experiment consisted of:

a control with a pop up with a shorter – to the point copy
a treatment with longer copy that articulated the “why’s” behind joining the community

Research Question

We believe that adding clear value articulation in the pop-up copy as to why you should join the community will achieve a greater number of sign-ups to the community (email acquisition) for returning visitors to the film pages.

Design

C: Shorter copy

T1: Longer copy w/ clear value articulation

Results

	Treatment Name	Conv. Rate	Relative Difference	Confidence	Average Gift
C:	Shorter copy	3.8%			$0.00
T1:	Longer copy w/ clear value articulation	4.6%	20.6%	73.8%	$0.00

This experiment has a required sample size of 4,988 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

This experiment ran for just over a month, and while the original goal was not statistically valid due to not enough returning visitors to see statistical significance and only a 73.8% Level of Confidence, there were some practical learnings we took away from this test. We did see over a 20% lift in emails acquired on the treatment, and week over week - we saw trends moving upwards to higher LoC's. While we cannot definitively say that the longer copy with greater value articulation was the only factor to increasing email signups to the community, we can say (based on numerous other similar experiments run for other NextAfter clients) that it would be worth continuing to test clear messaging strategies when trying to get a visitor to take a specific action. The key to strengthening value proposition includes the elements of appeal, exclusivity, clarity, and credibility. This experiment really focused the value proposition around the first three factors, in an effort to increase the strength of the message and the likelihood that a returning visitor would respond and take action. As the goal of the experiment was to see if we could increase email sign-ups to the community from returning visitors, a future test idea might be to acknowledge the returning visitor with language like "Welcome back" or "Nice to see you again", etc. Observations of the impact on other visitor segments in the experiment:

Mobile visitors had a 27.9% increase in email acquisition with a 93% level of confidence.
Email visitors had a 162.9% increase in email acquisition with a 99% level of confidence.

Experiment Documented by NextAfter

Question about experiment #87643

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.