NextAfter

How different audience motivations affect click rate in a Facebook ad (Part 2)

Experiment ID: #4616

NextAfter

Experiment Summary

Timeframe: 07/28/2016 - 08/01/2016

While setting up the advertising and promotions for an upcoming live broadcast, we wanted to determine if our various audience groups for our Facebook Ads would respond differently to various ad treatments. We have 3 main audience groups that we ran this test with: our housefile, housefile lookalikes, and nonprofit professionals.

Our original live broadcast Facebook Ad focused on the speaker and the title, but we thought people may be more motivated by different aspects of the focus topic. For the treatments, we developed an ad that focused on “Growing Your Donor File” and one that focused on “Empathy.”

Research Question

Which ad variation will the housefile lookalikes audience respond to more?

Design

C: Speaker
T1: Donor File
T2: Empathy

Results

 Treatment NameClick RateRelative DifferenceConfidence
C: Speaker 0.56%
T1: Donor File 0.60%6.9% 9.2%
T2: Empathy 1.3%124.6% 95.0%

This experiment has a required sample size of 2,160 in order to be valid. Since the experiment had a total sample size of 5,088, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    124.6% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

With this test on the housefile lookalikes audience, we found that the “Empathy” treatment outperformed the control and treatment 1, lifting click rate by 110%. This further confirms our learning from our test with the nonprofit professionals audience that it is more effective to focus our ads on the topic and results that will come from the live broadcast rather than the speaker.

This also shows us that even similar audience groups can have varying motivations: the “Empathy” treatment was the winner with this audience, but the “Donor File” treatment was the winner with the nonprofit professionals audience.


Experiment Documented by Nathan Hill
Nathan Hill is Vice President, NextAfter Institute.

Question about experiment #4616

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.