How targeted digital advertising affected direct mail results Experiment ID: #8421

Canadian Taxpayers Federation

A not-for-profit citizen's advocacy group dedicated to lower taxes, less waste and accountable government.

Experiment Summary

Timeframe: 10/26/2017 - 11/25/2017

The Canadian Taxpayers Federation wanted to test whether showing Facebook video ads to house file direct mail recipients, before and during the response period caused a lift in any key metric: conversion, donation size or total revenue. This test was based on the successful test executed by Hillsdale College.

The Taxpayers Federation created a control audience that was excluded from seeing any video ads, and a treatment audience that would see a rotation of three video ads (but no donation-centric or petition ads) for a one week period before the mail piece hit mailboxes and for three weeks after, the prime time in which a donor would respond. They chose videos that had been popular in the past, but were still relevant. They optimized the ads for reach—aiming to show ads to as large a percentage of the target group as possible. The goal of this test was to lift direct mail revenue—not to add an additional segment of online revenue or capture data through petitions. 

Ads were turned off once each video had an average frequency (number of times shown to each person reached) greater than 4.1. However, some videos were displayed as many as 4.5 times.

They spent just shy of $700 (CDN) to show ads to the treatment audience and waited to see the results come in.

Research Question

Will showing ads to a targeted group of direct mail recipients result in increased revenue from the treatment group?


C: Direct Mail Only
T1: Direct Mail plus Ads


Treatment Name Revenue per Visitor Relative Difference Confidence Average Gift
C: Direct Mail Only $6.32 $98.15
T1: Direct Mail plus Ads $7.93 25.4% 85.2% $117.55

This experiment has a required sample size of 42,949 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

It appears that with a 25.4% increase in dollars per visitor, at a 92.6% confidence level, that this was a success and should be expanded and replicated in the future. Overall, a $690 investment in Facebook ads, resulted in a nearly $10,000 increase in overall revenue.

Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.

Experiment Documented by...

Jeff Giddens

Jeff is the President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.