How a different messaging angle affects conversion rate from a Facebook ad Experiment ID: #19485

NextAfter

Experiment Summary

Timeframe: 1/1/2020 - 1/8/2020

At the start of the year, we launched a new Facebook ad campaign promoting the new State of Nonprofit Email Cultivation research report. As this is the first time to push the report in a focused way, we want to test a couple different messaging angles that each seemed to be a viable approach.

The control copy focused in on known stats about donor retention in order to a make the argument that something is broken with our cultivation strategies.

The treatment copy focused on the donor’s experience, illustrating how frustrated and jaded donors can feel if they are not cultivated well.

Each version ended up making the same call-to-action, using the same headline, image, etc. The primary change was the first 2-3 paragraphs of copy to set up the call-to-action.

Research Question

Which value proposition angle will lead to more report downloads?

Design

C: "Something is Broken" Value Proposition
T1: "Frustrated and Jaded Donors" Value Proposition

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: "Something is Broken" Value Proposition 0.40%
T1: "Frustrated and Jaded Donors" Value Proposition 1.1% 168.9% 100.0%

This experiment has a required sample size of 1,398 in order to be valid. Since the experiment had a total sample size of 7,911, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 168.9% increase in conversion rate
× 0% increase in average gift

Key Learnings

From this experiment, we learned that the “frustrated and jaded donors” approach is a more effective messaging approach – to the tune of a 168% increase in conversion over the control. It is interesting to see that data alone does not always move people to action. But using copy to make a more personal and human connection – and helping fundraisers to empathize with donors – can be a stronger copywriting method.

We’ll need to put this broader learning to the test in other scenarios to see if it holds true.


Experiment Documented by...

Nathan Hill

Nathan is an Optimization Evangelist at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.