19.7% lift How stripping out branding in an offer email affects clickthrough rate

Date Added: September 18, 2017
Research Partner: Heartlight Ministries
Element tested: Email Design

Heartlight Ministries was promoting their new free online course to their housefile. They had developed a heavily branded email template that, despite being signed by Mark Gregston (their founder), felt impersonal. They hypothesized that they could get a better response–measured in clickthrough rate–by stripping out much of the branding and sending an email that felt more personal in nature. They developed a plain-text treatment and split the active email file to determine a winner.



View the experiment »


-9.6% drop How adding an additional newsletter opt-in checkbox affects form completion rate

Date Added: September 18, 2017
Research Partner: The Heritage Foundation
Element tested: Name Acquisition Form

The Heritage Foundation was running an open letter to grow its email file and motivate the Senate to take action on judicial appointments. While this offer was running, they performed an analysis that found that subscribers to their Morning Bell were more likely to become donors. They decided to add a checkbox to the form to allow signers of the open letter to opt in to the Morning Bell. They hypothesized that it would decrease conversion, but the data showed that as long as the decrease was slight, the increased value of the added subscribers would more than make up for it. They launched an A/B test to determine a winner.



View the experiment »


-33.0% drop How a ‘Campaign Opt-Out’ Link Affects Email Unsubscribe Rate

Date Added: September 13, 2017
Research Partner: Dallas Theological Seminary
Element tested: Email Copy

Dallas Theological Seminary was running a campaign for North Texas Giving Day, a huge fundraising opportunity each year for North Texas nonprofits. As they planned for this campaign, they wanted to give their audience a way to remove themselves from further communication about North Texas Giving Day if they weren’t able to join this year. They wondered: If we allow individuals to ‘opt-out’ of a campaign, will the unsubscribe rate decrease? They created a treatment with one line of copy and a campaign ‘Opt-out’ link for viewers to click, and then they launched an A/B split test to find out.



View the experiment »


406.9% lift How Ad Congruency Affects Clickthrough Rate on a Facebook Ad

Date Added: September 12, 2017
Research Partner: Hillsdale College
Element tested: Advertising

Hillsdale College found in a previous test that adding course illustration improved clickthrough rate significantly, while increasing email acquisition rate by 85%. As they began to test this approach with other offers, they noticed that some of their offers have a variety of illustrations to choose from–their “Great Books 102” course offering, for example. Examining this particular course, they wondered: Will a course illustration that directly reinforces the authors mentioned in the ad copy increase clickthrough rate? They launched an A/B test on their ‘Great Books 102’ course to find out.



View the experiment »


39.1% lift How framing the audience for an offer affects conversion rate

Date Added: September 12, 2017
Research Partner: Buckner International
Element tested: Name Acquisition Headline, Name Acquisition Copy

Buckner International ran an offer focused on helping your children learn how to react to poverty. They wanted to know if refocusing the offer copy around the person receiving the ebook, rather than how their children would benefit, would impact conversion rate. The ads that preceded these landing pages also reflected this reframing of copy. The other page elements remained the same.



View the experiment »


Not Valid How the tone of an appeal affects clickthrough rate

Date Added: September 11, 2017
Research Partner: Buckner International
Element tested: Email Copy

Buckner International was sending an update on their Hurricane Harvey relief efforts with an appeal to raise more funds for their efforts. One email in the test took a “story” approach of a family affected, while the other email took a more urgent approach, with a direct ask for funds.



View the experiment »


81.4% lift How a direct ask on a blog impacts long-term donor conversion

Date Added: September 11, 2017
Research Partner: Alliance Defending Freedom
Element tested: Advertising

For Alliance Defending Freedom, the mobile version of their blog received a significant volume of traffic from Facebook. Historically, we had used this as an opportunity to acquire new email addresses to the file. We decided to test whether this audience could also have the potential to be a good donor acquisition source. To do this, we replaced the standard email acquisition offer with a direct ask for one of their clients, Barronelle.



View the experiment »


-72.0% drop How a Tailored, Remarketing Headline Affects Clickthrough Rate on a Facebook Ad

Date Added: September 11, 2017
Research Partner: Hillsdale College
Element tested: Advertising

Hillsdale College recently launched remarketing Facebook audiences, targeting individuals who viewed their course signup pages. They wondered if tailoring the Facebook headline to these visitors would increase clickthrough rate. They created a treatment with the headline “Begin your first class now »” and tested it against their control headline “Activate your free course now »”. They monitored this test for a little over a week and documented the results.



View the experiment »


-58.4% drop How the minimum premium gift impacts donor conversion

Date Added: September 11, 2017
Research Partner: The Heritage Foundation
Element tested: Advertising, Donation Page Copy

On one of their donor acquisition campaign, The Heritage Foundation was testing the use of a premium to increase conversion and average gift. Previous experiments had proven that the premium increased the likelihood of a donor to convert but we wanted to see if we could also increase to overall revenue coming from the campaign. We decided to try testing the premium at a $125 minimum gift level. The control treatment had the minimum gift set at $100.



View the experiment »


Not Valid How reduced copy and a more direct headline affect donor conversion

Date Added: September 10, 2017
Research Partner: Dallas Theological Seminary
Element tested: Name Acquisition Headline, Name Acquisition Copy

Dallas Theological Seminary offered a free online course called How to Read the Bible Like a Seminary Professor to allow their donors and prospects to experience the value proposition of the Seminary. Although they had a good donor conversion rate, they wanted to see if they could improve email acquisition by making the value proposition on the signup page more appealing.

They created a treatment with a more direct headline: “Learn to read the Bible like never before”. Then, they removed the bulleted list from the copy to reduce the overall length of the copy, hypothesizing that it was not core to the value proposition on the page.

They split traffic to the page and launched an A/B test to determine a winner.



View the experiment »