The Heritage Foundation

How using a more provocative title for the subject of a survey impacts Facebook ad clicks

Experiment ID: #194746

The Heritage Foundation

Experiment Summary

Timeframe: 12/19/2024 - 01/08/2025

The Heritage Foundation was running a “Stop the Administrative State” survey and promoting through Facebook ads. In an attempt to increase click rates on the ads, they wanted to test whether using the more provocative term “Deep State” in place of the more straightforward “Administrative State” survey title would achieve this effect because people would be more provoked to take action against the subject when referred to as a more negative term.

The control’s Facebook ad image and copy referred to the Administrative State, while the treatment’s ad image and copy referred to the Deep State, and both received 50% of traffic. In addition, the control led to survey landing page with “Administrative State” language, and the treatment led to a survey page with “Deep State” language for continuity.

This test ran for 3 weeks.

Research Question

We believe that using an more provocative name for the Administrative State for survey Facebook ad viewers will achieve an increase in clicks on the ad because the term “Deep State” invokes stronger emotions. because the term “Deep State” invokes stronger emotions.

Design

C: Control
Array ( [0] => Array ( [ID] => 69657 [id] => 69657 [title] => Control.png [filename] => Control-23.png [filesize] => 192320 [url] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23.png [link] => https://www.nextafter.com/experiments/how-using-a-more-provocative-title-for-the-subject-of-a-survey-impacts-facebook-ad-clicks/attachment/control-png-9726/ [alt] => [author] => 0 [description] => [caption] => [name] => control-png-9726 [status] => inherit [uploaded_to] => 69616 [date] => 2025-01-21 16:10:18 [modified] => 2025-01-21 16:10:18 [menu_order] => 0 [mime_type] => image/png [type] => image [subtype] => png [icon] => https://nextafter-1a91a.kxcdn.com/wp-includes/images/media/default.png [width] => 324 [height] => 946 [sizes] => Array ( [thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-150x150.png [thumbnail-width] => 150 [thumbnail-height] => 150 [medium] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-103x300.png [medium-width] => 103 [medium-height] => 300 [medium_large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23.png [medium_large-width] => 324 [medium_large-height] => 946 [large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23.png [large-width] => 324 [large-height] => 946 [1536x1536] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23.png [1536x1536-width] => 324 [1536x1536-height] => 946 [2048x2048] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23.png [2048x2048-width] => 324 [2048x2048-height] => 946 [ab-block-post-grid-landscape] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-600x400.png [ab-block-post-grid-landscape-width] => 600 [ab-block-post-grid-landscape-height] => 400 [ab-block-post-grid-square] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-600x600.png [ab-block-post-grid-square-width] => 600 [ab-block-post-grid-square-height] => 600 [tp-image-grid] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-700x700.png [tp-image-grid-width] => 700 [tp-image-grid-height] => 700 [post-thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-150x150.png [post-thumbnail-width] => 150 [post-thumbnail-height] => 150 [admin] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-17x50.png [admin-width] => 17 [admin-height] => 50 [full] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23.png [full-width] => 324 [full-height] => 946 [thumb_372] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23.png [thumb_372-width] => 324 [thumb_372-height] => 946 [single_post_thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-214x625.png [single_post_thumbnail-width] => 214 [single_post_thumbnail-height] => 625 [thumb_32_32] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-11x32.png [thumb_32_32-width] => 11 [thumb_32_32-height] => 32 [thumb_48_48] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-16x48.png [thumb_48_48-width] => 16 [thumb_48_48-height] => 48 [thumb_64_64] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-22x64.png [thumb_64_64-width] => 22 [thumb_64_64-height] => 64 [thumb_128_128] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-44x128.png [thumb_128_128-width] => 44 [thumb_128_128-height] => 128 [thumb_122_67] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-23x67.png [thumb_122_67-width] => 23 [thumb_122_67-height] => 67 [thumb_153_153] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-52x153.png [thumb_153_153-width] => 52 [thumb_153_153-height] => 153 [thumb_130_163] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-56x163.png [thumb_130_163-width] => 56 [thumb_130_163-height] => 163 [thumb_232_177] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-61x177.png [thumb_232_177-width] => 61 [thumb_232_177-height] => 177 [thumb_272_325] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-223x650.png [thumb_272_325-width] => 223 [thumb_272_325-height] => 650 [thumb_271_346] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-119x346.png [thumb_271_346-width] => 119 [thumb_271_346-height] => 346 [thumb_271_271] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-93x271.png [thumb_271_271-width] => 93 [thumb_271_271-height] => 271 [thumb_272_172] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-59x172.png [thumb_272_172-width] => 59 [thumb_272_172-height] => 172 [thumb_372_461] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-158x461.png [thumb_372_461-width] => 158 [thumb_372_461-height] => 461 [thumb_372_500] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-171x500.png [thumb_372_500-width] => 171 [thumb_372_500-height] => 500 [thumb_385_270] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-92x270.png [thumb_385_270-width] => 92 [thumb_385_270-height] => 270 [thumb_416_241] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-83x241.png [thumb_416_241-width] => 83 [thumb_416_241-height] => 241 [thumb_448_410] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-140x410.png [thumb_448_410-width] => 140 [thumb_448_410-height] => 410 [thumb_448_410_retina] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-281x820.png [thumb_448_410_retina-width] => 281 [thumb_448_410_retina-height] => 820 [thumb_574_382] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-131x382.png [thumb_574_382-width] => 131 [thumb_574_382-height] => 382 [thumb_776_395] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-135x395.png [thumb_776_395-width] => 135 [thumb_776_395-height] => 395 ) ) [1] => Array ( [ID] => 69660 [id] => 69660 [title] => "Deep_State".png [filename] => Deep_State-8.png [filesize] => 199316 [url] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8.png [link] => https://www.nextafter.com/experiments/how-using-a-more-provocative-title-for-the-subject-of-a-survey-impacts-facebook-ad-clicks/attachment/deep_state-png-9/ [alt] => [author] => 0 [description] => [caption] => [name] => deep_state-png-9 [status] => inherit [uploaded_to] => 69616 [date] => 2025-01-21 16:10:26 [modified] => 2025-01-21 16:10:26 [menu_order] => 0 [mime_type] => image/png [type] => image [subtype] => png [icon] => https://nextafter-1a91a.kxcdn.com/wp-includes/images/media/default.png [width] => 340 [height] => 955 [sizes] => Array ( [thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-150x150.png [thumbnail-width] => 150 [thumbnail-height] => 150 [medium] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-107x300.png [medium-width] => 107 [medium-height] => 300 [medium_large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8.png [medium_large-width] => 340 [medium_large-height] => 955 [large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8.png [large-width] => 340 [large-height] => 955 [1536x1536] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8.png [1536x1536-width] => 340 [1536x1536-height] => 955 [2048x2048] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8.png [2048x2048-width] => 340 [2048x2048-height] => 955 [ab-block-post-grid-landscape] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-600x400.png [ab-block-post-grid-landscape-width] => 600 [ab-block-post-grid-landscape-height] => 400 [ab-block-post-grid-square] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-600x600.png [ab-block-post-grid-square-width] => 600 [ab-block-post-grid-square-height] => 600 [tp-image-grid] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-700x700.png [tp-image-grid-width] => 700 [tp-image-grid-height] => 700 [post-thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-150x150.png [post-thumbnail-width] => 150 [post-thumbnail-height] => 150 [admin] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-18x50.png [admin-width] => 18 [admin-height] => 50 [full] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8.png [full-width] => 340 [full-height] => 955 [thumb_372] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8.png [thumb_372-width] => 340 [thumb_372-height] => 955 [single_post_thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-223x625.png [single_post_thumbnail-width] => 223 [single_post_thumbnail-height] => 625 [thumb_32_32] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-11x32.png [thumb_32_32-width] => 11 [thumb_32_32-height] => 32 [thumb_48_48] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-17x48.png [thumb_48_48-width] => 17 [thumb_48_48-height] => 48 [thumb_64_64] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-23x64.png [thumb_64_64-width] => 23 [thumb_64_64-height] => 64 [thumb_128_128] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-46x128.png [thumb_128_128-width] => 46 [thumb_128_128-height] => 128 [thumb_122_67] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-24x67.png [thumb_122_67-width] => 24 [thumb_122_67-height] => 67 [thumb_153_153] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-54x153.png [thumb_153_153-width] => 54 [thumb_153_153-height] => 153 [thumb_130_163] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-58x163.png [thumb_130_163-width] => 58 [thumb_130_163-height] => 163 [thumb_232_177] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-63x177.png [thumb_232_177-width] => 63 [thumb_232_177-height] => 177 [thumb_272_325] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-231x650.png [thumb_272_325-width] => 231 [thumb_272_325-height] => 650 [thumb_271_346] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-123x346.png [thumb_271_346-width] => 123 [thumb_271_346-height] => 346 [thumb_271_271] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-96x271.png [thumb_271_271-width] => 96 [thumb_271_271-height] => 271 [thumb_272_172] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-61x172.png [thumb_272_172-width] => 61 [thumb_272_172-height] => 172 [thumb_372_461] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-164x461.png [thumb_372_461-width] => 164 [thumb_372_461-height] => 461 [thumb_372_500] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-178x500.png [thumb_372_500-width] => 178 [thumb_372_500-height] => 500 [thumb_385_270] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-96x270.png [thumb_385_270-width] => 96 [thumb_385_270-height] => 270 [thumb_416_241] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-86x241.png [thumb_416_241-width] => 86 [thumb_416_241-height] => 241 [thumb_448_410] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-146x410.png [thumb_448_410-width] => 146 [thumb_448_410-height] => 410 [thumb_448_410_retina] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-292x820.png [thumb_448_410_retina-width] => 292 [thumb_448_410_retina-height] => 820 [thumb_574_382] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-136x382.png [thumb_574_382-width] => 136 [thumb_574_382-height] => 382 [thumb_776_395] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-141x395.png [thumb_776_395-width] => 141 [thumb_776_395-height] => 395 ) ) )
T1: "Deep State"
Array ( [0] => Array ( [ID] => 69657 [id] => 69657 [title] => Control.png [filename] => Control-23.png [filesize] => 192320 [url] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23.png [link] => https://www.nextafter.com/experiments/how-using-a-more-provocative-title-for-the-subject-of-a-survey-impacts-facebook-ad-clicks/attachment/control-png-9726/ [alt] => [author] => 0 [description] => [caption] => [name] => control-png-9726 [status] => inherit [uploaded_to] => 69616 [date] => 2025-01-21 16:10:18 [modified] => 2025-01-21 16:10:18 [menu_order] => 0 [mime_type] => image/png [type] => image [subtype] => png [icon] => https://nextafter-1a91a.kxcdn.com/wp-includes/images/media/default.png [width] => 324 [height] => 946 [sizes] => Array ( [thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-150x150.png [thumbnail-width] => 150 [thumbnail-height] => 150 [medium] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-103x300.png [medium-width] => 103 [medium-height] => 300 [medium_large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23.png [medium_large-width] => 324 [medium_large-height] => 946 [large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23.png [large-width] => 324 [large-height] => 946 [1536x1536] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23.png [1536x1536-width] => 324 [1536x1536-height] => 946 [2048x2048] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23.png [2048x2048-width] => 324 [2048x2048-height] => 946 [ab-block-post-grid-landscape] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-600x400.png [ab-block-post-grid-landscape-width] => 600 [ab-block-post-grid-landscape-height] => 400 [ab-block-post-grid-square] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-600x600.png [ab-block-post-grid-square-width] => 600 [ab-block-post-grid-square-height] => 600 [tp-image-grid] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-700x700.png [tp-image-grid-width] => 700 [tp-image-grid-height] => 700 [post-thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-150x150.png [post-thumbnail-width] => 150 [post-thumbnail-height] => 150 [admin] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-17x50.png [admin-width] => 17 [admin-height] => 50 [full] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23.png [full-width] => 324 [full-height] => 946 [thumb_372] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23.png [thumb_372-width] => 324 [thumb_372-height] => 946 [single_post_thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-214x625.png [single_post_thumbnail-width] => 214 [single_post_thumbnail-height] => 625 [thumb_32_32] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-11x32.png [thumb_32_32-width] => 11 [thumb_32_32-height] => 32 [thumb_48_48] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-16x48.png [thumb_48_48-width] => 16 [thumb_48_48-height] => 48 [thumb_64_64] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-22x64.png [thumb_64_64-width] => 22 [thumb_64_64-height] => 64 [thumb_128_128] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-44x128.png [thumb_128_128-width] => 44 [thumb_128_128-height] => 128 [thumb_122_67] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-23x67.png [thumb_122_67-width] => 23 [thumb_122_67-height] => 67 [thumb_153_153] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-52x153.png [thumb_153_153-width] => 52 [thumb_153_153-height] => 153 [thumb_130_163] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-56x163.png [thumb_130_163-width] => 56 [thumb_130_163-height] => 163 [thumb_232_177] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-61x177.png [thumb_232_177-width] => 61 [thumb_232_177-height] => 177 [thumb_272_325] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-223x650.png [thumb_272_325-width] => 223 [thumb_272_325-height] => 650 [thumb_271_346] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-119x346.png [thumb_271_346-width] => 119 [thumb_271_346-height] => 346 [thumb_271_271] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-93x271.png [thumb_271_271-width] => 93 [thumb_271_271-height] => 271 [thumb_272_172] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-59x172.png [thumb_272_172-width] => 59 [thumb_272_172-height] => 172 [thumb_372_461] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-158x461.png [thumb_372_461-width] => 158 [thumb_372_461-height] => 461 [thumb_372_500] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-171x500.png [thumb_372_500-width] => 171 [thumb_372_500-height] => 500 [thumb_385_270] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-92x270.png [thumb_385_270-width] => 92 [thumb_385_270-height] => 270 [thumb_416_241] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-83x241.png [thumb_416_241-width] => 83 [thumb_416_241-height] => 241 [thumb_448_410] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-140x410.png [thumb_448_410-width] => 140 [thumb_448_410-height] => 410 [thumb_448_410_retina] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-281x820.png [thumb_448_410_retina-width] => 281 [thumb_448_410_retina-height] => 820 [thumb_574_382] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-131x382.png [thumb_574_382-width] => 131 [thumb_574_382-height] => 382 [thumb_776_395] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-23-135x395.png [thumb_776_395-width] => 135 [thumb_776_395-height] => 395 ) ) [1] => Array ( [ID] => 69660 [id] => 69660 [title] => "Deep_State".png [filename] => Deep_State-8.png [filesize] => 199316 [url] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8.png [link] => https://www.nextafter.com/experiments/how-using-a-more-provocative-title-for-the-subject-of-a-survey-impacts-facebook-ad-clicks/attachment/deep_state-png-9/ [alt] => [author] => 0 [description] => [caption] => [name] => deep_state-png-9 [status] => inherit [uploaded_to] => 69616 [date] => 2025-01-21 16:10:26 [modified] => 2025-01-21 16:10:26 [menu_order] => 0 [mime_type] => image/png [type] => image [subtype] => png [icon] => https://nextafter-1a91a.kxcdn.com/wp-includes/images/media/default.png [width] => 340 [height] => 955 [sizes] => Array ( [thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-150x150.png [thumbnail-width] => 150 [thumbnail-height] => 150 [medium] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-107x300.png [medium-width] => 107 [medium-height] => 300 [medium_large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8.png [medium_large-width] => 340 [medium_large-height] => 955 [large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8.png [large-width] => 340 [large-height] => 955 [1536x1536] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8.png [1536x1536-width] => 340 [1536x1536-height] => 955 [2048x2048] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8.png [2048x2048-width] => 340 [2048x2048-height] => 955 [ab-block-post-grid-landscape] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-600x400.png [ab-block-post-grid-landscape-width] => 600 [ab-block-post-grid-landscape-height] => 400 [ab-block-post-grid-square] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-600x600.png [ab-block-post-grid-square-width] => 600 [ab-block-post-grid-square-height] => 600 [tp-image-grid] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-700x700.png [tp-image-grid-width] => 700 [tp-image-grid-height] => 700 [post-thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-150x150.png [post-thumbnail-width] => 150 [post-thumbnail-height] => 150 [admin] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-18x50.png [admin-width] => 18 [admin-height] => 50 [full] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8.png [full-width] => 340 [full-height] => 955 [thumb_372] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8.png [thumb_372-width] => 340 [thumb_372-height] => 955 [single_post_thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-223x625.png [single_post_thumbnail-width] => 223 [single_post_thumbnail-height] => 625 [thumb_32_32] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-11x32.png [thumb_32_32-width] => 11 [thumb_32_32-height] => 32 [thumb_48_48] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-17x48.png [thumb_48_48-width] => 17 [thumb_48_48-height] => 48 [thumb_64_64] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-23x64.png [thumb_64_64-width] => 23 [thumb_64_64-height] => 64 [thumb_128_128] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-46x128.png [thumb_128_128-width] => 46 [thumb_128_128-height] => 128 [thumb_122_67] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-24x67.png [thumb_122_67-width] => 24 [thumb_122_67-height] => 67 [thumb_153_153] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-54x153.png [thumb_153_153-width] => 54 [thumb_153_153-height] => 153 [thumb_130_163] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-58x163.png [thumb_130_163-width] => 58 [thumb_130_163-height] => 163 [thumb_232_177] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-63x177.png [thumb_232_177-width] => 63 [thumb_232_177-height] => 177 [thumb_272_325] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-231x650.png [thumb_272_325-width] => 231 [thumb_272_325-height] => 650 [thumb_271_346] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-123x346.png [thumb_271_346-width] => 123 [thumb_271_346-height] => 346 [thumb_271_271] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-96x271.png [thumb_271_271-width] => 96 [thumb_271_271-height] => 271 [thumb_272_172] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-61x172.png [thumb_272_172-width] => 61 [thumb_272_172-height] => 172 [thumb_372_461] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-164x461.png [thumb_372_461-width] => 164 [thumb_372_461-height] => 461 [thumb_372_500] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-178x500.png [thumb_372_500-width] => 178 [thumb_372_500-height] => 500 [thumb_385_270] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-96x270.png [thumb_385_270-width] => 96 [thumb_385_270-height] => 270 [thumb_416_241] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-86x241.png [thumb_416_241-width] => 86 [thumb_416_241-height] => 241 [thumb_448_410] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-146x410.png [thumb_448_410-width] => 146 [thumb_448_410-height] => 410 [thumb_448_410_retina] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-292x820.png [thumb_448_410_retina-width] => 292 [thumb_448_410_retina-height] => 820 [thumb_574_382] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-136x382.png [thumb_574_382-width] => 136 [thumb_574_382-height] => 382 [thumb_776_395] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Deep_State-8-141x395.png [thumb_776_395-width] => 141 [thumb_776_395-height] => 395 ) ) )

Results

 Treatment NameClick RateRelative DifferenceConfidence
C: Control 3.3%
T1: "Deep State" 3.3%-1.1% 15.9%

This experiment has a required sample size of 1,973,450 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

The key learning from this experiment is that using the term “Deep State” in place of “Administrative State” did not lead to an increase in clicks on the Facebook ad. In fact, the treatment group saw a slight 1.1% decrease in Clicks for All Traffic compared to the control group.

However, since the statistical level of confidence is only 15.9%, these results are not considered valid and should not be used for future decision making.

In addition to click rate, the treatment saw a valid 12.8% decrease in sign up rate on the landing page at a 97% level of confidence. This suggests that the term “Deep State” may have led to fewer people completing the survey.

Finally, the treatment saw a 8.5% decrease in donor acquisition rate on the post-survey landing page compared to the control, but at a 18.6% level of confidence, this result cannot be relied upon as valid.

In future experimentation, it is essential to carefully design the study with a larger sample size or run the experiment for a longer period of time to increase the statistical power. Additionally, it may be beneficial to test different variables or messaging strategies to determine what resonates best with the audience. Conducting thorough data analysis and ensuring statistical significance will help in making informed decisions based on experimental results.


Experiment Documented by NextAfter

Question about experiment #194746

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.