171.4% lift How creating a simpler email format increases donations and revenue

Date Added: June 22, 2020 Research Partner: The Missionary Oblates of Mary Immaculate Element tested: Email Design, Email Copy

Missionary Oblates send out a monthly evangelize newsletter. They use the simpler format on all other emails that come from their organization as a way of creating a one-to-one relationship with the subscribers. They want to see if this same concept would apply to the stylized newsletter sent out each month. To do this, instead of breaking up all the stories, they put the entire first story into the email followed by the other links. The goal was the measure the impact on engagement (including giving).

View the experiment »


-43.4% drop How optimizing for Subscriptions on Facebook impacts recurring donor acquisition

Date Added: June 22, 2020 Research Partner: Care Net Element tested: Advertising

Roughly a year ago, Care Net ran an experiment to determine the impact of using different optimization methodologies within Facebook. In that experiment, we learned that optimizing our Facebook campaigns to acquire “Purchasers” yielded more donors than optimizing for ”Custom Conversions” (aka emails) or even for optimizing for “Donors” as Facebook defines them. Our hypothesis for this result is that Facebook’s advertising algorithm incorporated online buying behavior with ”Purchase” optimization that the other metrics did not. Ultimately, this experiment drastically improved acquisition efforts and significantly lowered costs. Recently, Facebook rolled out additional conversion options that are available for optimization. One of those options is a ”Subscription” which is intended to represent an ongoing monthly payment. Given Care Net’s emphasis on acquiring recurring donors, we decided to run an experiment to see if this metric could acquire more recurring donors than “Purchase” optimization.

View the experiment »


-43.4% drop How optimizing for Subscriptions on Facebook impacts recurring donor acquisition

Date Added: June 22, 2020 Research Partner: Care Net Element tested: Advertising

Roughly a year ago, Care Net ran an experiment to determine the impact of using different optimization methodologies within Facebook. In that experiment, we learned that optimizing our Facebook campaigns to acquire “Purchasers” yielded more donors than optimizing for ”Custom Conversions” (aka emails) or even for optimizing for “Donors” as Facebook defines them. Our hypothesis for this result is that Facebook’s advertising algorithm incorporated online buying behavior with ”Purchase” optimization that the other metrics did not. Ultimately, this experiment drastically improved acquisition efforts and significantly lowered costs. Recently, Facebook rolled out additional conversion options that are available for optimization. One of those options is a ”Subscription” which is intended to represent a ongoing monthly payment. Given Care Net’s emphasis on acquiring recurring donors, we decided to run an experiment to see if this metric could acquire more recurring donors than “Purchase” optimization.

View the experiment »


17.2% lift How emphasizing the unknown impacts open rate

Date Added: June 22, 2020 Research Partner: EWTN Element tested: Email Subject Line

EWTN was sending out an email as part of their calendar year end campaign. As with every email, we wanted to find a way to increase the engagement of the email audience. With this email, we decided to test two different subject lines. Both emphasized the "unknown" to try to increase intrigue but the control focused on the "work" and the treatment mentioned a "lesser known gift." We wanted to see which would increase opens and clicks.

View the experiment »


-28.3% drop How a message box on the front end of a donation ask affects conversion

Date Added: June 22, 2020 Research Partner: CaringBridge Element tested: Donation Page Form

CaringBridge's tribute widget was one of the primary channels of donations on their site. Appearing at the bottom of every person's homepage and journal, the tribute widget made a donation ask, and then gave the user the option to enter a message of support to go along with their gift. The CaringBridge team had always wanted to know whether this was a "feature or a bug". Did the message box improve conversion by getting more people to start their gift by writing a message—or did the large box intimidate potential donors and turn them away? They decided to simply remove the box and see what happened. Without it, there was just a standalone donation ask with a button. They ran the treatment to just 20% of the audience to mitigate potential risk, since this got enough traffic to validate with a small window of time.

View the experiment »


-68.3% drop How the way a content offer is presented at the end of an article impacts name conversion

Date Added: June 22, 2020 Research Partner: Focus on the Family Element tested: Advertising

On the Focus on the Family website, there is a content offer at the end of each article to people to sign-up for a free video series. The video series is relevant to the content they are reading, uses a picture to show them who the teacher of the video series is, communicates what the video series will help them with, and has a button where people can be taken to a sign-up page to get the offer. This had been running on the site for a while and we hypothesized that we might be able to get more sign-ups by presenting the offer as a sticky bar at the bottom of the page instead of an inline article offer.

View the experiment »


20.3% lift How visually changing the “Give Monthly” button impacts traffic to the donation page

Date Added: June 22, 2020 Research Partner: Focus on the Family Element tested: Advertising

On the Focus on the Family website, they present two options for people to give, "Donate" and "Give Monthly". We hypothesized that we could increase traffic to the monthly donation page by visually setting off the "Give Monthly" button. To do this, we created a subtle animation on the "Give Monthly" button that used a heart icon that pulsed once and the effect would then ripple across the rest of the button. We split the traffic 50/50 and ran this experiment across the website's entire navigation.

View the experiment »


-36.0% drop How visually presenting the donate button differently impacts traffic to the donation page

Date Added: June 22, 2020 Research Partner: Care Net Element tested: Advertising

On Care Net's primary navigation, they provide a link that says "Donate". This link takes people to their donation page where people can go and make a gift to support the organization. We hypothesized that presenting the link as a button and as an animated button might increase visual presence and ultimately increase traffic to the donation page resulting in more gifts and revenue. To test this, we developed two variations. The first variation was simply making the donate link, look like a button. For the second variation, we added subtle animation to the button that had the heart of the Care Net logo pulse once and the effect ripple across the rest of the button.

View the experiment »


-6.7% drop How the time of the send impacts engagement

Date Added: June 22, 2020 Research Partner: Back to the Bible Element tested: Email Sender

Back to the Bible was sending out an email to their housefile promoting their latest free resource. They wanted to see if an evening send would have any change on impact when compared to a send during the middle of the day.

View the experiment »


Not Valid How campaign first messaging impacts clicks from donors in Giving Tuesday Now emails

Date Added: June 22, 2020 Research Partner: EWTN Element tested: Email Copy

On Giving Tuesday Now, EWTN ran a campaign focused around acquiring recurring donors. Because Giving Tuesday Now is new to people, we were curious if that message would resonate with donors and nondonors on EWTN's email file. We tested this by using mission based messaging at the beginning of the email and then later talked about Giving Tuesday Now for the control, and developed a treatment used Giving Tuesday Now focused messaging at the beginning of the email and then later used missional messaging. We tested this with two different segments - donors and nondonors to see if there was a difference in response based on audience segment as well.

View the experiment »