How a humanized pop-up affected email sign ups
SCPR
Experiment Summary
Timeframe: 11/23/2022 - 11/28/2023
Southern California Public Radio is a public radio station based out of Las Angeles. A popular newsletter that SCPR offers is “How to LA” which provides a daily morning email with current events, upcoming events, things to do, and anything someone may want to know about LA.
We have seen success with other organizations in testing personal-human invites for newsletter sign-ups or donations.
“How to LA”‘s sender is a well-known person within the SCPR audience so we decided to test with this specific newsletter first.
Research Question
We believe that by using a human invitation for website visitors will achieve higher email acquisition rate.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.17% | ||
| T1: | Humanized Popup | 0.23% | 35.8% | 95.9% |
This experiment has a required sample size of 40,775 in order to be valid. Since the experiment had a total sample size of 88,873, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 35.8% increase in conversion rate
× 0% increase in average gift
Key Learnings
This experiment ran for 5 days and we saw a valid lift with a 35.8% increase in email acquisition with a 95.9% level of confidence.
Our next phases of testing will be in testing clarity around what the newsletter provides, length in copy, and CTA. Stay tuned!