How using a different sender increased open rates
Americans for Prosperity
Experiment Summary
Ended On: 03/14/2020
Americans for Prosperity has been using two senders to communicate to their house file for slightly more than a year now. Tim, the President of the organization, is the primary solicitor and has historically sent 1-2 weekly emails to promote news and updates to the reader. In the summer of 2019, we added an additional sender who largely cultivates the audience (doesn’t ask) other than in the context of a high urgency campaign (as a follow-up to Tim’s earlier solicitation emails if they haven’t yet responded before the deadline).
We wanted to specifically quantify how the audience responded to the two senders by running an A/B split test on a weekly cultivation email.
Research Question
How does our audience respond to our various senders?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 7.9% | ||
| T1: | Treatment #1 | 9.1% | 15.0% | 100.0% |
This experiment has a required sample size of 4,241 in order to be valid. Since the experiment had a total sample size of 48,933, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
15.0% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The experiment validated that the non-soliciting sender achieved a +15% increase in open rate with 99.9% level of confidence.
Beyond the fact that “Matt” (the non-soliciting sender in the program) makes far less fundraising appeals via email than Tim (the primary solicitor in the program), we observed one other note: Matt’s email domain was also different than Tim’s.
Further experimentation is required to specifically identify whether or not the sender domain reputation or the less-likely-to-solicit sender’s name (Matt) in the treatment was what increased open rates. Our theory, though, is that Tim’s email sender domain reputation is lower than the email domain reputation from the email Matt used. For this reason, we expect a higher number of emails to go into the SPAM folder from Tim’s email, which could be a contributing factor to his decrease in open rate.
Finally, it’s also worth noting that even though Matt received a 15% increase in open rate, this sender only achieved a +7.7% increase in article clickthrough rate, but it did not validate with a 44% level of confidence.
What could this mean? It could mean that the audience has been conditioned to expect Tim to send them a link and ask them to do something—and those that open the email are twice as likely to take Tim’s call to action when compared to the less-likely-to-solicit sender (Matt).
Again, further experimentation is required to specifically isolate the sender and the email domain to better understand the learning from this experiment and how best to use this information to improve results moving forward.