How we can validate the impact in changing the sender | NextAfter
Alliance Defending Freedom

How we can validate the impact in changing the sender

Experiment ID: #18479

Alliance Defending Freedom

Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.

Experiment Summary

Ended On: 10/08/2019

Alliance Defending Freedom had by running a 5 month long cultivation experiment where half of their email file received weekly cultivation sends from an ADF staff member (Sarah.) Due to Sarah going on leave due to maternity, we ended up having to change the sender. In the week prior, Maureen took over for Sarah after being introduced the week before. In this first send, we saw the lowest open rate we had seen with this entire experiment.

This lower open rate was cause for concern since the entire purpose of these emails is to get engagement with the file. We decided to run a subject line experiment to see if we can find a subject that would get the level of engagement we were use to seeing with these emails.

Research Question

Will a straight-forward subject line (Latest attack on freed) get better engagement than one that is more sensational (Political Correctness run amuck)?


C: Political correctness run amuck
T1: Latest attack on freedom


Treatment Name Open Rate Relative Difference Confidence
C: Political correctness run amuck 16.9%
T1: Latest attack on freedom 18.9% 11.8% 100.0%

This experiment has a required sample size of 2,834 in order to be valid. Since the experiment had a total sample size of 58,783, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    11.8% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

This experiment proved two things for us:

  1. Maureen is able to get the same level of engagement as we are use to seeing with Sarah as the sender.
  2. The more straight-forward subject line received an 11.8% increase in email engagement.

Experiment Documented by NextAfter

Question about experiment #18479

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.