The Heritage Foundation

How a simpler, more direct value proposition affects clickthrough rate on a Facebook ad

Experiment ID: #2805

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Timeframe: 03/12/2017 - 04/10/2017

The shorter ad with the direct question produced a 122.1% increase in clickthrough rate, which showed that the elements of this ad increased the clarity of the value proposition. Longer copy can be more effective, but only if if clarifies the value proposition. In addition, the direct question to provoke the reader to respond by taking the intended action.

This success prompted further testing to isolate the variables of length and the direct question to understand how each of these played into the success of the ad.

Research Question

How does a shorter, more direct value proposition affect clickthrough rate on a Facebook ad?

Design

C: ad2 - long copy
T1: ad1

Results

 Treatment NameClick RateRelative DifferenceConfidence
C: ad2 - long copy 1.1%
T1: ad1 2.4%122.1% 100.0%

This experiment has a required sample size of 891 in order to be valid. Since the experiment had a total sample size of 1,097,283, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    122.1% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The Heritage Foundation was promoting a quiz around the “Gadsden Flag” that led to a membership offer with the flag as a premium. They had created two versions of a Facebook ad, as they had seen some research that longer, more engaging copy could increase clickthrough and conversion rate. The control, in this case, had much longer copy that talked about the history of the flag and made an offer to the reader to test their knowledge. The treatment copy was much shorter, and asked a direct question to the reader: “How much do you know about the Gadsden Flag?”

They split the traffic to determine which ad attracted more clicks.


Experiment Documented by NextAfter

Question about experiment #2805

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.