35.3% lift How Creating Mystery in the Subject Line can Influence Conversion Rates

Date Added: October 9, 2017
Research Partner: NextAfter
Element tested: Email Subject Line

When setting up this email to promote an upcoming year-end webinar, we wondered if placing the percent difference between “#GivingTuesday” and the “last week of the calendar year” in the subject line was giving away too much too soon.

By moving the percentage to the body of the email, rather than giving away the result of our analysis in the subject line, we thought we could create more mystery and drive a greater desire to open the email.



View the experiment »


-36.9% drop How adding a phone number field to an acquisition form impacts conversion

Date Added: October 4, 2017
Research Partner: Care Net
Element tested: Name Acquisition Form

Care Net offers an online course called, Pro-Life 101. When someone signs up for the course, the form has fields where they provide personal information. Getting a phone number from a new name is very valuable to the organization. They wanted to test if the conversion rate of the page would be impacted if they asked for a person’s phone number.



View the experiment »


115.9% lift How the tone of advertising impacts donor conversion

Date Added: October 3, 2017
Research Partner: The Heritage Foundation
Element tested: Advertising

The Heritage Foundation had learned from previous experiments that a homepage callout section that focused on new member acquisition as able to drive a high volume of donor motivated traffic. In previous tests on this callout, we had tested the entire funnel and found that a membership offer was the highest performing with regards to new donors but that the “Drain the Swamp” message resulted in the most clicks.

We decided to test out a “Drain the Swamp” message on the homepage callout against the standard “Join Heritage” language. Both of these would include a final call to action of “Become a member” and would land on the same donation landing page that focused on membership as the offer.



View the experiment »


Not Valid How an ‘Up Next’ slideout affects email acquisition rate on Desktop

Date Added: October 2, 2017
Research Partner: Alliance Defending Freedom
Element tested: Engagement

Alliance Defending Freedom recently conducted an experiment  on mobile devices, where they asked: Is it possible to increase clickthrough rate by providing an ‘Up Next’ article? The result was a valid 29% increase in traffic without a valid decrease in conversion. They wondered if they would see the same results if they conducted this experiment with desktop users as well. They launched a desktop-only test to find out.



View the experiment »


Not Valid How urgency on a donation page affects donor conversion rate

Date Added: October 2, 2017
Research Partner: Buckner International
Element tested: Donation Page Copy

Buckner was testing more urgent copy for an immediate need on the donation page of their content offer, 7 Scriptures to Pray for the Child in your Life. The immediate need focused around Hurricane Harvey and the Buckner staff and families affected by the disaster. The ads and acquisition page for the offer remained the same.



View the experiment »



154.5% lift How targeting direct mail prospects with digital advertising affected direct mail revenue

Date Added: October 1, 2017
Research Partner: Hillsdale College
Element tested: Advertising

Hillsdale College had significant investments in their direct mail program and wanted to see if they could optimize the return by investing in digital marketing that would target these prospects in the time period that they received a direct mail letter.

They created a control audience that was excluded from seeing any ads, and a treatment audience that would see a rotation of brand and course offering ads (but no donation-centric ads) for a two week period before the mail piece hit mailboxes and for two weeks after, the prime time in which a donor would respond. They optimized the ads for reach—aiming to show ads to as large a percentage of the target group as possible. The goal of this test was to lift direct mail revenue—not to add an additional segment of online revenue. 

They spent just shy of $1,000 to show ads to the treatment audience and waited to see the results come in.



View the experiment »


-12.5% drop How a personalized note affects clickthrough rate on a Facebook ad

Date Added: September 27, 2017
Research Partner: Hillsdale College
Element tested: Advertising

Hillsdale College’s course, “An Introduction to C.S. Lewis: Writings and Significance” has remained one of their most popular courses over the last few years. After receiving a significant amount of positive feedback, they found that many people wanted to own a DVD version of this course. So they created a DVD Box Set and began to offer it to their fans online. As they began this new campaign, they wondered: Will a personal note from Dr. Larry Arnn increase the motivation and clickthrough rate of the ad viewer? They created a treatment and launched an A/B test to find out.



View the experiment »


16.6% lift How increasing the financial impact of the gift affected donor conversion

Date Added: September 25, 2017
Research Partner: The Heritage Foundation
Element tested: Donation Page Copy

The Heritage Foundation had a major donor step forward and agree to cover all of the administrative and fundraising costs for the organization for a period of time. The goal behind this action was to allow 100% of all other donations to be directed toward the mission. It was our hypothesis that this type of language would increase the perceived impact of the donors’ gifts which would increase their likelihood to donate.



View the experiment »


69.2% lift Gmail’s handling of full-format vs. low-format emails

Date Added: September 19, 2017
Research Partner: Jewish Voice Ministries
Element tested: Email Design

We had already been testing a low-format version of our emails (no tables, no images, limited CSS, visible links) and seeing that they performed several percentage points better in open rate than the fully formatted versions. We suspected that one or more email clients was treating low-format emails differently than full-format emails, and our first suspect was Google due to its tabbed inbox system.



View the experiment »