e3 Partners

IAS – General Email – March 2021

Experiment ID: #63518

e3 Partners

Experiment Summary

Ended On: 06/09/2021

Control was a fully designed, normal Newsletter format coming from I Am Second.

Treatment #1 was a completely plain-text and personalized Newsletter coming from Doug Bender

Research Question

We believe that changing the sender and using plain text for our general list will achieve a higher open rate.

Design

C: Control
T1: Treatment #1

Results

 Treatment NameOpen RateRelative DifferenceConfidence
C: Control 17.3%
T1: Treatment #1 25.7%48.9% 100.0%

This experiment has a required sample size of 182 in order to be valid. Since the experiment had a total sample size of 31,123, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    48.9% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

Treatment #1 increased the open rate, as was hypothesized. Click-through rate was higher in the fully-designed email. We moved forward by testing a combination of a plain-text email with more visuals against the control in the next General newsletter.


Experiment Documented by NextAfter

Question about experiment #63518

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.