IAS – General Email – February 2021
e3 Partners
Experiment Summary
Timeframe: 02/23/2021 - 03/09/2021
Control was our normal, fully designed newsletter format from I Am Second.
Treatment #1 was a fully plain-text email coming from Doug.
Research Question
We believe that changing sender and using plain-text for our general list will achieve a higher open rate.
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 25.7% | ||
| T1: | Treatment #1 | 42.8% | 66.6% | 100.0% |
This experiment has a required sample size of 59 in order to be valid. Since the experiment had a total sample size of 22,590, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
66.6% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
Open rate was higher on Treatment #1, but Click-through late was much higher on the control. We will continue testing this to see if it changes.