41.5% lift How additional cultivation impacts online giving

Date Added: October 31, 2019 Research Partner: Alliance Defending Freedom Element tested: Engagement, Email Copy

During a data analysis performed for ADF at the beginning of the calendar year, we discovered a handful of trends related to the email file that were concerning:

  • A significant portion ADF subscribers had not opened an email within the last six months
  • New and existing donors were more likely to disengage as non-donor subscribers
  • ADF was sending twice as many solicitation emails as it was cultivation and stewardship-focused emails. Simply put: the subscribers were being asked twice as much as they were given back to.
While the digital fundraising program had been growing, these trends illustrated a potential problem in the near future if they were not corrected. The historical success of ADF's online fundraising was closely correlated to the size and engagement of the email file.   Given the ambitious fundraising goals, we couldn't reduce the number of solicitations. So, instead, we decided to test out an additional cultivation series that would increase the number of emails sent to the file. The additional emails would focus on educating and feeding back into the various segments. There would be no additional solicitations. At a tactical level, this email series consisted of simply-designed emails sent each Friday by one of the authors on the blog (Sarah K). They would include a link to a blog post or a video and we emphasized getting replies to the email. To validate the impact of this series, we split the email file in half, paying special attention to ensure that there was equal representation of active, lapsed and non-donors and that the average revenue donor was similar between the splits.

View the experiment »


77.2% lift How a simplified newsletter affects click rate over a templatized version

Date Added: October 28, 2019 Research Partner: Dallas Theological Seminary Element tested: Email Copy

DTS was sending out their top three DTS Voice articles to their subscribers as a cultivation opportunity, and wanted to see whether a simplified template (links only in bullet points) or a more designed layout, with imagery and CTA's along with blurbs about the content, performed better. They used the same content pieces in each email.

View the experiment »


84.2% lift How a simplified newsletter affects click rate over a templatized version, part II

Date Added: October 28, 2019 Research Partner: Dallas Theological Seminary Element tested: Email Copy

In a previous test, DTS was sending out their top three DTS Voice articles to their subscribers as a cultivation opportunity, and wanted to see whether a simplified template (links only in bullet points) or a more designed layout, with imagery and CTA's along with blurbs about the content, performed better. They used the same content pieces in each email. After the original test, we hypothesized that that the CTA buttons were potentially causing friction more than the design of the newsletter, and that that was the cause of the original lift from the second version. In this test, we tried linked text in the templatized version vs. the original simplified template.

View the experiment »


11.8% lift How we can validate the impact in changing the sender

Date Added: October 28, 2019 Research Partner: Alliance Defending Freedom Element tested: Email Subject Line

Alliance Defending Freedom had by running a 5 month long cultivation experiment where half of their email file received weekly cultivation sends from an ADF staff member (Sarah.) Due to Sarah going on leave due to maternity, we ended up having to change the sender. In the week prior, Maureen took over for Sarah after being introduced the week before. In this first send, we saw the lowest open rate we had seen with this entire experiment. This lower open rate was cause for concern since the entire purpose of these emails is to get engagement with the file. We decided to run a subject line experiment to see if we can find a subject that would get the level of engagement we were use to seeing with these emails.

View the experiment »


14.7% lift How changing the subject line of Evangelize emails to a name of a story increases opens

Date Added: September 22, 2019 Research Partner: The Missionary Oblates of Mary Immaculate Element tested: Email Subject Line

The monthly newsletter sent out by MAMI uses a standard subject line each month that clearly identifies the publication and the date it is sent. However, there was the hypothesis that if we made the subject line more casual, we could increase the open rate. We decided to test this with a 50/50 split.

View the experiment »


13.0% lift How minimal content impacts subscriber engagement

Date Added: September 22, 2019 Research Partner: Alliance Defending Freedom Element tested: Email Copy

The primary goal of Alliance Defending Freedom's weekly cultivation series is to engage subscribers with content that is central to ADF's mission. With this in mind, past optimization efforts have centered around finding the optimal tactics to lift open and click through rates. All of the emails are sent from the same sender, "Sarah", and are written to appear as one-to-one as possible. In recent discussions, a hypothesis was proposed that we could potentially increase the number of visitors clicking by reducing the amount of content we put into the email. The theory was based upon the thought that we may be spending too much time "selling" the content instead of letting the visitors get to the information. To test this hypothesis, we created a treatment version of the weekly email with a dramatically simplified message.

View the experiment »



26.8% lift How the kind and tone of cultivation emails sent can impact donor conversion

Date Added: August 16, 2019 Research Partner: Care Net Element tested: Email Copy

Care Net sends a weekly cultivation email to donors and nondonors on their file. They call this email their "weekly update" and it usually includes a couple of interesting articles or podcast episodes they encourage people to read or listen to. It's sent every Saturday morning from Ardee Coolidge - a familiar sender to their file as he is also normally the sender of appeal emails. We know that cultivation is critical to the success of an online fundraising program, but we questioned if we made it more personal in nature and changed elements of what was being sent, if we could increase engagement and ultimately donor conversion of their email file. 

To test this, we split their file in half - having an equal number of donors and nondonors in each segment, and ran a longitudinal test over a three month period that ended during their fiscal year-end. The control segment continued to receive the standard cultivation/communication and the treatment segment received similar content, but the approach was slightly different. We created a more intentional, high-touch email that used personal tone copy and presented people with one piece of content to read or listen to. We also changed the sender of the treatment emails over the three month duration of the experiment.



View the experiment »


Not Valid How leading with a question impact email click rate

Date Added: July 23, 2019 Research Partner: Alliance Defending Freedom Element tested: Email Copy

Alliance Defending Freedom was sending out one of their weekly cultivation emails as part of a longitudinal experiment to improve email engagement. In the previous week, we had observed a surprising result in an experiment. When starting the email with a personal anecdote, we saw a decreased level of engagement. We had a hypothesis that this may be due to the way the two emails differed in their introductions. The control started with the question to the reader while the treatment started with the story. Since there were several other factors that could have led to the observed result, we decided to isolate the question factor and run another A/B split. The only difference in the control and treatments email would then be whether they begin with a question or a statement.

View the experiment »


-36.2% drop How adding a personal story impacted email engagement

Date Added: July 22, 2019 Research Partner: Alliance Defending Freedom Element tested: Email Copy

Alliance Defending Freedom was continuing the longitudinal experiment to increase cultivation to a targeted segment of their email house file. The goal of these emails was to simply engage the audience and provide them with ongoing content and stewardship. With that in mind, we wanted to find ways to increase that engagement. One hypothesis we developed was that we could potentially increase the personal-feel of these emails by including stories from the sender's own life. We believed this personal touch might increase the likelihood that someone would click on the email.

View the experiment »