How a humanized pop-up affected email sign ups

Experiment ID: #116015


Experiment Summary

Timeframe: 11/23/2022 - 11/28/2023

Southern California Public Radio is a public radio station based out of Las Angeles. A popular newsletter that SCPR offers is “How to LA” which provides a daily morning email with current events, upcoming events, things to do, and anything someone may want to know about LA.

We have seen success with other organizations in testing personal-human invites for newsletter sign-ups or donations.

“How to LA”‘s sender is a well-known person within the SCPR audience so we decided to test with this specific newsletter first.

Research Question

We believe that by using a human invitation for website visitors will achieve higher email acquisition rate.


C: Control
T1: Humanized Popup


 Treatment NameConv. RateRelative DifferenceConfidence
C: Control 0.17%
T1: Humanized Popup 0.23%35.8% 95.9%

This experiment has a required sample size of 40,775 in order to be valid. Since the experiment had a total sample size of 88,873, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 35.8% increase in conversion rate
× 0% increase in average gift

Key Learnings

This experiment ran for 5 days and we saw a valid lift with a 35.8% increase in email acquisition with a 95.9% level of confidence.

Our next phases of testing will be in testing clarity around what the newsletter provides, length in copy, and CTA. Stay tuned!

Experiment Documented by NextAfter

Question about experiment #116015

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.