How changing the sending domain impacted email engagement rates Experiment ID: #21860

Americans for Prosperity

Experiment Summary

Timeframe: 7/14/2020 - 7/19/2020

Americans for Prosperity had been running a number of petition and survey acquisition offers and produced hundreds of thousands of new email addresses, donors, and revenue through this acquisition process. However, with such a large volume of emails coming through these types of offers, over time they noticed their email engagement rates (deliveries, opens, and clicks) going down.

They had two email sender domains available to them, so instead of sending from the one they’ve been primarily using, they decided to test sending an email from the secondary email sender domain to monitor email engagement rates against the primary domain.

Research Question

Will switching email sender domains increase email engagement rates?

Design

C: "Tim Phillips"
T1: "Tim Phillips"

Results

Treatment Name Open Rate Relative Difference Confidence
C: "Tim Phillips" 7.4%
T1: "Tim Phillips" 10.1% 37.2% 100.0%

This experiment has a required sample size of 817 in order to be valid. Since the experiment had a total sample size of 70,086, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    37.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

With a level of confidence of 100% — we observed a 37.2% increase in email open rates when using the secondary sender domain.

Subsequently, we observed a +49.8% increase in email clickthrough rate for the email used within the test, as well.

If email deliverability is an issue for you, perhaps creating a new sender domain may be a technique you can use to increase email deliverability and engagement over time.

When creating a new email sender domain, it’s “reputation” must be built up over time. To do this, it requires that the list be segmented into two types: (1) engaged emails, and (2) disengaged emails. For at least 3-months, this new sender domain should only send emails to the “engaged” list. This will help it establish itself as being of “good quality” when evaluated by mail servers.

This will lead to good inbox rates for all emails sent over time.

While this “warm-up” period of time is underway, continue to send your emails from the original sender domain, but only to the “disengaged” portion of the list.

You can send the same email creative, but it’s advised that you lead with emailing the “engaged” file first, wait at least an hour, then send the same email to the “disengaged” list from the original email sender domain.

Ideally, recent opens/clicks from the “disengaged” list should move those contacts to the “engaged” segment (and vise versa for people who continue not to open/click if they were originally placed into the “engaged” list).

All of this will allow you to warm up a different email sender domain and could yield increased results in email delivery, email inboxing rates, which will subsequently lead to increases in open and click rates, as well.


Experiment Documented by...

Greg Colunga

Greg is Executive Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.