30.1% lift How “need-based” copy with a quantified ask affects donor conversion during a matching challenge

Date Added: March 23, 2017
Research Partner: CaringBridge
Element tested: Donation Page Copy

CaringBridge had an area on their journals called the “tribute widget”. This widget asked the reader to honor their friend or loved one by making a donation in their honor. This had caused some clarity issues around who was receiving the gift, but testing and optimization had lowered the refund rate significantly.

They were entering a 48-hour matching challenge and wanted to test two new factors in the tribute widget. First, they wanted to make a more direct ask. Rather than asking the reader to “honor” their friend, they wanted to ask them to “help power [their] site”. Additionally, they wanted to include a quantified ask—giving the donor the handle of “$30 powers their site for one month”.

They set up an A/B test during the match period to see how it affected results.



View the experiment »

—…

Read More

13.9% lift How reducing friction through a two-step form increases email acquisition rate

Date Added: March 22, 2017
Research Partner: Hillsdale College
Element tested: Name Acquisition Design, Name Acquisition Form

Hillsdale College had previously verified that a two-step signup form for their online courses increased email acquisition rate. However, the platform on which they had built this two-step form made it difficult to do further testing on the signup process. So they decided to rebuild the two-step form on a new platform that made future testing easier.

They wanted to re-test the concept to make sure that the lift still held—and they also wanted to test a new concept. The previous winning treatment had two unique pages that each contained a part of the full form. Their new concept animated the two halves of the form on a single page in a smooth motion that didn’t require a second page.

They launched a three-way split URL test to validate a winning treatment.



View the experiment »


-23.1% drop How a subject line’s perceived impact on the subscriber influences click through rate

Date Added: March 21, 2017
Research Partner: Illinois Policy Institute
Element tested: Email Subject Line

In an effort to educate their subscriber base on the public’s sentiment toward the state’s current budget crisis, the Illinois Policy Institute sent an email with the results of an independent 3rd party poll. The results of the poll aligned with the message IPI had been promoting since the current session began. It also supported the budget plan the the organization had developed and was working with state legislators to pass.

The goal of the email was to educate and engage the subscriber file so we decided to test out two subject lines that highlighted the contents of the message: “tax hikes” and “overspending.”



View the experiment »


-55.4% drop How a specific giving challenge affects donor conversion

Date Added: March 20, 2017
Research Partner: Alliance Defending Freedom
Element tested: Email Copy

Alliance Defending Freedom has an active legal case defending a florist named Barronelle Stutzman, who is being sued for running her business according to her religious beliefs.

In an effort to raise additional funds beyond their known donor base, they utilized some rented email lists to reach a new audience. As part of the communication, they wanted to help potential donors understand what the organization was up against. The ACLU, the organization that was helping to sue Barronelle, had raised 24 million dollars in a single weekend.  To counter the well-funded opposition, they issued a challenge for one million donors to give $24 or more.

However, there was concern about the potential mental friction of donors understanding the math and whether the challenge may reduce the average gift amount. We decided to run a test to find out.



View the experiment »


63.9% lift How straightforward copy and a quantified ask affects donor conversion

Date Added: March 20, 2017
Research Partner: CaringBridge
Element tested: Donation Page Copy

CaringBridge had run an experiment that implemented more “need-based”, common language on their tribute widget, which was the start of the giving process on their journal pages. They had run this experiment as part of a match, so they wanted to re-run the experiment without the matching language to see if that impacted conversion.

The control asked visitors to “honor” their friend or loved one with a donation. The treatment used much more straightforward language: “Can you help power their site?” It also incorporated a quantified ask: “$30 keeps their site online for a month.”

They set up an A/B test to determine a winner in a non-match scenario.



View the experiment »


26.7% lift How additional value proposition copy affects donor conversion

Date Added: March 20, 2017
Research Partner: CaringBridge
Element tested: Email Copy

CaringBridge sent a fundraising email to their housefile as part of a monthly appeal. The email told a personal story from a mother who had experienced the power of CaringBridge firsthand. Their control email told a compelling personal story using a convincing personal tone, but they thought it lacked the value proposition copy necessary to compel readers to give.

They created a treatment that increased the amount of copy significantly, but provided (what they perceived to be) a stronger value proposition. They believed that even though this increased the time it took to read the email, that the additions would produce a greater return.

They split their file in half and launched an experiment to determine a winner.



View the experiment »


360.0% lift How different imagery affects the conversion rate from a Facebook ad

Date Added: March 20, 2017
Research Partner: NextAfter
Element tested: Advertising

When launching a new research study, we wanted to ensure that we were getting the most out of our Facebook advertising.

Our control used typical imagery with a picture of the eBook, and text on the ad saying “Get the Free eBook.”

We developed a treatment with the hypothesis that more people might download the study if the ad introduced the nature of the content, rather than making an overt offer right in the news feed.

A 2nd treatment was developed with the hypothesis that an infographic could more accurately convey the value of the research than the book’s artwork, prompting more downloads.



View the experiment »


20.5% lift How third-party credibility affects email acquisition rate

Date Added: March 20, 2017
Research Partner: Hillsdale College
Element tested: Name Acquisition Design, Name Acquisition Copy

Hillsdale College had launched a course on the United States Supreme Court. Previous testing had already shown that showing a prospect the course schedule outperformed an image of two radio hosts that often promote the course. They had gathered some student testimonials that they wanted to test in place of the course schedule. They hypothesized that these testimonials might outperform the course schedule. Even though the course schedule gave more insight into what was on the other side, the testimonials gave third-party credibility to how good the content was on the other side.

They launched an A/B test to determine a winner.



View the experiment »


146.9% lift How clarifying the value proposition through inductive copy affects donor conversion

Date Added: March 17, 2017
Research Partner: Heritage Action for America
Element tested: Donation Page Copy

Heritage Action for America was running a petition campaign to confirm Supreme Court nominee Neil Gorsuch. After the petition was signed, the signer was presented with a donation ask. Though they were acquiring thousands of signatures, donor conversion was very low, so they wanted to launch an experiment to increase response. Upon review of the page, they noticed that the headline was relatively self-contained—meaning that it provided confirmation of the signature and did not entice the reader to read any more of the copy.

They created a treatment that still thanked the visitor for signing, but immediately teased them by offering “three reasons” why their signature was critical, to keep them reading down the page. They hypothesized that if they could capture the reader’s attention for just a second longer, they could increase donor conversion by drawing the reader into the body copy.

They launched an A/B test to determine a winner.



View the experiment »


80.8% lift How including an image in a personal email affects donor conversion

Date Added: March 16, 2017
Research Partner: Alliance Defending Freedom
Element tested: Email Design

Alliance Defending Freedom was raising money for a very specific case. They were renting email lists for the first time, and so they knew that, to many of the recipients, they were introducing the person they were defending for the first time. However, since this case made national news, they knew that many of the recipients would be familiar with the case, if not the name of the defendant.

They had tested several different treatments and had arrived at a very plain text email with no formatting. However, since they were introducing someone new, they thought that it might produce a lift if they included a picture of Barronelle, the defendant. Since this was truly an introduction, then they thought the reader might be more likely to respond if they could see her before they reached the landing page, and when they reached the landing page, they might be more likely to convert.

They created a treatment with an image and split the next email list rental in half to determine a winner.



View the experiment »