How we can validate the impact in changing the sender
Alliance Defending Freedom
Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.
Experiment Summary
Ended On: 10/08/2019
Alliance Defending Freedom had by running a 5 month long cultivation experiment where half of their email file received weekly cultivation sends from an ADF staff member (Sarah.) Due to Sarah going on leave due to maternity, we ended up having to change the sender. In the week prior, Maureen took over for Sarah after being introduced the week before. In this first send, we saw the lowest open rate we had seen with this entire experiment.
This lower open rate was cause for concern since the entire purpose of these emails is to get engagement with the file. We decided to run a subject line experiment to see if we can find a subject that would get the level of engagement we were use to seeing with these emails.
Research Question
Will a straight-forward subject line (Latest attack on freed) get better engagement than one that is more sensational (Political Correctness run amuck)?
Design
Results
Treatment Name | Open Rate | Relative Difference | Confidence | |
---|---|---|---|---|
C: | Political correctness run amuck | 16.9% | ||
T1: | Latest attack on freedom | 18.9% | 11.8% | 100.0% |
This experiment has a required sample size of 2,834 in order to be valid. Since the experiment had a total sample size of 58,783, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
11.8% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
This experiment proved two things for us:
- Maureen is able to get the same level of engagement as we are use to seeing with Sarah as the sender.
- The more straight-forward subject line received an 11.8% increase in email engagement.
Question about experiment #23732
If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.