The Heritage Foundation

How using a more provocative title for the subject of a survey impacts Google ad clicks

Experiment ID: #194779

The Heritage Foundation

Experiment Summary

Ended On: 01/21/2025

The Heritage Foundation was running a “Stop the Administrative State” survey and promoting through Google ads. In an attempt to increase click rates on the ads, they wanted to test whether using the more provocative term “Deep State” in place of the more straightforward “Administrative State” survey title would achieve this effect because people would be more provoked to take action against the subject when referred to as a more negative term.

The control’s Google ad image and copy options, including 8 headline and 5 description options, referred to the Administrative State, while the treatment’s ad image and copy referred to the Deep State, and both received 50% of traffic. In addition, the control led to survey landing page with “Administrative State” language, and the treatment led to a survey page with “Deep State” language for continuity.

This test ran for 3 weeks.

Research Question

We believe that using an more provocative name for the Administrative State for survey Google ad viewers will achieve an increase in clicks on the ad because the term “Deep State” invokes stronger emotions.

Design

C: Control
Array ( [0] => Array ( [ID] => 69670 [id] => 69670 [title] => Control.png [filename] => Control-29.png [filesize] => 108494 [url] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [link] => https://www.nextafter.com/experiments/how-using-a-more-provocative-title-for-the-subject-of-a-survey-impacts-google-ad-clicks/attachment/control-png-9733/ [alt] => [author] => 0 [description] => [caption] => [name] => control-png-9733 [status] => inherit [uploaded_to] => 69661 [date] => 2025-01-21 16:25:14 [modified] => 2025-01-21 16:25:14 [menu_order] => 0 [mime_type] => image/png [type] => image [subtype] => png [icon] => https://nextafter-1a91a.kxcdn.com/wp-includes/images/media/default.png [width] => 313 [height] => 260 [sizes] => Array ( [thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-150x150.png [thumbnail-width] => 150 [thumbnail-height] => 150 [medium] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-300x249.png [medium-width] => 300 [medium-height] => 249 [medium_large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [medium_large-width] => 313 [medium_large-height] => 260 [large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [large-width] => 313 [large-height] => 260 [1536x1536] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [1536x1536-width] => 313 [1536x1536-height] => 260 [2048x2048] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [2048x2048-width] => 313 [2048x2048-height] => 260 [ab-block-post-grid-landscape] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-600x400.png [ab-block-post-grid-landscape-width] => 600 [ab-block-post-grid-landscape-height] => 400 [ab-block-post-grid-square] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-600x600.png [ab-block-post-grid-square-width] => 600 [ab-block-post-grid-square-height] => 600 [tp-image-grid] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-700x700.png [tp-image-grid-width] => 700 [tp-image-grid-height] => 700 [post-thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-150x150.png [post-thumbnail-width] => 150 [post-thumbnail-height] => 150 [admin] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-50x42.png [admin-width] => 50 [admin-height] => 42 [full] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [full-width] => 313 [full-height] => 260 [thumb_372] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [thumb_372-width] => 313 [thumb_372-height] => 260 [single_post_thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [single_post_thumbnail-width] => 313 [single_post_thumbnail-height] => 260 [thumb_32_32] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-32x27.png [thumb_32_32-width] => 32 [thumb_32_32-height] => 27 [thumb_48_48] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-48x40.png [thumb_48_48-width] => 48 [thumb_48_48-height] => 40 [thumb_64_64] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-64x53.png [thumb_64_64-width] => 64 [thumb_64_64-height] => 53 [thumb_128_128] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-128x106.png [thumb_128_128-width] => 128 [thumb_128_128-height] => 106 [thumb_122_67] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-81x67.png [thumb_122_67-width] => 81 [thumb_122_67-height] => 67 [thumb_153_153] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-153x127.png [thumb_153_153-width] => 153 [thumb_153_153-height] => 127 [thumb_130_163] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-130x108.png [thumb_130_163-width] => 130 [thumb_130_163-height] => 108 [thumb_232_177] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-213x177.png [thumb_232_177-width] => 213 [thumb_232_177-height] => 177 [thumb_272_325] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [thumb_272_325-width] => 313 [thumb_272_325-height] => 260 [thumb_271_346] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-271x225.png [thumb_271_346-width] => 271 [thumb_271_346-height] => 225 [thumb_271_271] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-271x225.png [thumb_271_271-width] => 271 [thumb_271_271-height] => 225 [thumb_272_172] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-207x172.png [thumb_272_172-width] => 207 [thumb_272_172-height] => 172 [thumb_372_461] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [thumb_372_461-width] => 313 [thumb_372_461-height] => 260 [thumb_372_500] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [thumb_372_500-width] => 313 [thumb_372_500-height] => 260 [thumb_385_270] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [thumb_385_270-width] => 313 [thumb_385_270-height] => 260 [thumb_416_241] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-290x241.png [thumb_416_241-width] => 290 [thumb_416_241-height] => 241 [thumb_448_410] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [thumb_448_410-width] => 313 [thumb_448_410-height] => 260 [thumb_448_410_retina] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [thumb_448_410_retina-width] => 313 [thumb_448_410_retina-height] => 260 [thumb_574_382] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [thumb_574_382-width] => 313 [thumb_574_382-height] => 260 [thumb_776_395] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [thumb_776_395-width] => 313 [thumb_776_395-height] => 260 ) ) [1] => Array ( [ID] => 69671 [id] => 69671 [title] => Treatment_#1.png [filename] => Treatment_1-13.png [filesize] => 104425 [url] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [link] => https://www.nextafter.com/experiments/how-using-a-more-provocative-title-for-the-subject-of-a-survey-impacts-google-ad-clicks/attachment/treatment_1-png-3460/ [alt] => [author] => 0 [description] => [caption] => [name] => treatment_1-png-3460 [status] => inherit [uploaded_to] => 69661 [date] => 2025-01-21 16:25:23 [modified] => 2025-01-21 16:25:23 [menu_order] => 0 [mime_type] => image/png [type] => image [subtype] => png [icon] => https://nextafter-1a91a.kxcdn.com/wp-includes/images/media/default.png [width] => 314 [height] => 268 [sizes] => Array ( [thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-150x150.png [thumbnail-width] => 150 [thumbnail-height] => 150 [medium] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-300x256.png [medium-width] => 300 [medium-height] => 256 [medium_large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [medium_large-width] => 314 [medium_large-height] => 268 [large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [large-width] => 314 [large-height] => 268 [1536x1536] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [1536x1536-width] => 314 [1536x1536-height] => 268 [2048x2048] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [2048x2048-width] => 314 [2048x2048-height] => 268 [ab-block-post-grid-landscape] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-600x400.png [ab-block-post-grid-landscape-width] => 600 [ab-block-post-grid-landscape-height] => 400 [ab-block-post-grid-square] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-600x600.png [ab-block-post-grid-square-width] => 600 [ab-block-post-grid-square-height] => 600 [tp-image-grid] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-700x700.png [tp-image-grid-width] => 700 [tp-image-grid-height] => 700 [post-thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-150x150.png [post-thumbnail-width] => 150 [post-thumbnail-height] => 150 [admin] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-50x43.png [admin-width] => 50 [admin-height] => 43 [full] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [full-width] => 314 [full-height] => 268 [thumb_372] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [thumb_372-width] => 314 [thumb_372-height] => 268 [single_post_thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [single_post_thumbnail-width] => 314 [single_post_thumbnail-height] => 268 [thumb_32_32] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-32x27.png [thumb_32_32-width] => 32 [thumb_32_32-height] => 27 [thumb_48_48] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-48x41.png [thumb_48_48-width] => 48 [thumb_48_48-height] => 41 [thumb_64_64] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-64x55.png [thumb_64_64-width] => 64 [thumb_64_64-height] => 55 [thumb_128_128] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-128x109.png [thumb_128_128-width] => 128 [thumb_128_128-height] => 109 [thumb_122_67] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-79x67.png [thumb_122_67-width] => 79 [thumb_122_67-height] => 67 [thumb_153_153] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-153x131.png [thumb_153_153-width] => 153 [thumb_153_153-height] => 131 [thumb_130_163] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-130x111.png [thumb_130_163-width] => 130 [thumb_130_163-height] => 111 [thumb_232_177] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-207x177.png [thumb_232_177-width] => 207 [thumb_232_177-height] => 177 [thumb_272_325] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [thumb_272_325-width] => 314 [thumb_272_325-height] => 268 [thumb_271_346] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-271x231.png [thumb_271_346-width] => 271 [thumb_271_346-height] => 231 [thumb_271_271] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-271x231.png [thumb_271_271-width] => 271 [thumb_271_271-height] => 231 [thumb_272_172] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-202x172.png [thumb_272_172-width] => 202 [thumb_272_172-height] => 172 [thumb_372_461] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [thumb_372_461-width] => 314 [thumb_372_461-height] => 268 [thumb_372_500] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [thumb_372_500-width] => 314 [thumb_372_500-height] => 268 [thumb_385_270] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [thumb_385_270-width] => 314 [thumb_385_270-height] => 268 [thumb_416_241] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-282x241.png [thumb_416_241-width] => 282 [thumb_416_241-height] => 241 [thumb_448_410] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [thumb_448_410-width] => 314 [thumb_448_410-height] => 268 [thumb_448_410_retina] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [thumb_448_410_retina-width] => 314 [thumb_448_410_retina-height] => 268 [thumb_574_382] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [thumb_574_382-width] => 314 [thumb_574_382-height] => 268 [thumb_776_395] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [thumb_776_395-width] => 314 [thumb_776_395-height] => 268 ) ) )
T1: Treatment #1
Array ( [0] => Array ( [ID] => 69670 [id] => 69670 [title] => Control.png [filename] => Control-29.png [filesize] => 108494 [url] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [link] => https://www.nextafter.com/experiments/how-using-a-more-provocative-title-for-the-subject-of-a-survey-impacts-google-ad-clicks/attachment/control-png-9733/ [alt] => [author] => 0 [description] => [caption] => [name] => control-png-9733 [status] => inherit [uploaded_to] => 69661 [date] => 2025-01-21 16:25:14 [modified] => 2025-01-21 16:25:14 [menu_order] => 0 [mime_type] => image/png [type] => image [subtype] => png [icon] => https://nextafter-1a91a.kxcdn.com/wp-includes/images/media/default.png [width] => 313 [height] => 260 [sizes] => Array ( [thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-150x150.png [thumbnail-width] => 150 [thumbnail-height] => 150 [medium] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-300x249.png [medium-width] => 300 [medium-height] => 249 [medium_large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [medium_large-width] => 313 [medium_large-height] => 260 [large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [large-width] => 313 [large-height] => 260 [1536x1536] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [1536x1536-width] => 313 [1536x1536-height] => 260 [2048x2048] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [2048x2048-width] => 313 [2048x2048-height] => 260 [ab-block-post-grid-landscape] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-600x400.png [ab-block-post-grid-landscape-width] => 600 [ab-block-post-grid-landscape-height] => 400 [ab-block-post-grid-square] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-600x600.png [ab-block-post-grid-square-width] => 600 [ab-block-post-grid-square-height] => 600 [tp-image-grid] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-700x700.png [tp-image-grid-width] => 700 [tp-image-grid-height] => 700 [post-thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-150x150.png [post-thumbnail-width] => 150 [post-thumbnail-height] => 150 [admin] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-50x42.png [admin-width] => 50 [admin-height] => 42 [full] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [full-width] => 313 [full-height] => 260 [thumb_372] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [thumb_372-width] => 313 [thumb_372-height] => 260 [single_post_thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [single_post_thumbnail-width] => 313 [single_post_thumbnail-height] => 260 [thumb_32_32] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-32x27.png [thumb_32_32-width] => 32 [thumb_32_32-height] => 27 [thumb_48_48] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-48x40.png [thumb_48_48-width] => 48 [thumb_48_48-height] => 40 [thumb_64_64] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-64x53.png [thumb_64_64-width] => 64 [thumb_64_64-height] => 53 [thumb_128_128] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-128x106.png [thumb_128_128-width] => 128 [thumb_128_128-height] => 106 [thumb_122_67] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-81x67.png [thumb_122_67-width] => 81 [thumb_122_67-height] => 67 [thumb_153_153] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-153x127.png [thumb_153_153-width] => 153 [thumb_153_153-height] => 127 [thumb_130_163] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-130x108.png [thumb_130_163-width] => 130 [thumb_130_163-height] => 108 [thumb_232_177] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-213x177.png [thumb_232_177-width] => 213 [thumb_232_177-height] => 177 [thumb_272_325] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [thumb_272_325-width] => 313 [thumb_272_325-height] => 260 [thumb_271_346] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-271x225.png [thumb_271_346-width] => 271 [thumb_271_346-height] => 225 [thumb_271_271] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-271x225.png [thumb_271_271-width] => 271 [thumb_271_271-height] => 225 [thumb_272_172] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-207x172.png [thumb_272_172-width] => 207 [thumb_272_172-height] => 172 [thumb_372_461] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [thumb_372_461-width] => 313 [thumb_372_461-height] => 260 [thumb_372_500] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [thumb_372_500-width] => 313 [thumb_372_500-height] => 260 [thumb_385_270] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [thumb_385_270-width] => 313 [thumb_385_270-height] => 260 [thumb_416_241] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29-290x241.png [thumb_416_241-width] => 290 [thumb_416_241-height] => 241 [thumb_448_410] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [thumb_448_410-width] => 313 [thumb_448_410-height] => 260 [thumb_448_410_retina] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [thumb_448_410_retina-width] => 313 [thumb_448_410_retina-height] => 260 [thumb_574_382] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [thumb_574_382-width] => 313 [thumb_574_382-height] => 260 [thumb_776_395] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-29.png [thumb_776_395-width] => 313 [thumb_776_395-height] => 260 ) ) [1] => Array ( [ID] => 69671 [id] => 69671 [title] => Treatment_#1.png [filename] => Treatment_1-13.png [filesize] => 104425 [url] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [link] => https://www.nextafter.com/experiments/how-using-a-more-provocative-title-for-the-subject-of-a-survey-impacts-google-ad-clicks/attachment/treatment_1-png-3460/ [alt] => [author] => 0 [description] => [caption] => [name] => treatment_1-png-3460 [status] => inherit [uploaded_to] => 69661 [date] => 2025-01-21 16:25:23 [modified] => 2025-01-21 16:25:23 [menu_order] => 0 [mime_type] => image/png [type] => image [subtype] => png [icon] => https://nextafter-1a91a.kxcdn.com/wp-includes/images/media/default.png [width] => 314 [height] => 268 [sizes] => Array ( [thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-150x150.png [thumbnail-width] => 150 [thumbnail-height] => 150 [medium] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-300x256.png [medium-width] => 300 [medium-height] => 256 [medium_large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [medium_large-width] => 314 [medium_large-height] => 268 [large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [large-width] => 314 [large-height] => 268 [1536x1536] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [1536x1536-width] => 314 [1536x1536-height] => 268 [2048x2048] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [2048x2048-width] => 314 [2048x2048-height] => 268 [ab-block-post-grid-landscape] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-600x400.png [ab-block-post-grid-landscape-width] => 600 [ab-block-post-grid-landscape-height] => 400 [ab-block-post-grid-square] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-600x600.png [ab-block-post-grid-square-width] => 600 [ab-block-post-grid-square-height] => 600 [tp-image-grid] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-700x700.png [tp-image-grid-width] => 700 [tp-image-grid-height] => 700 [post-thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-150x150.png [post-thumbnail-width] => 150 [post-thumbnail-height] => 150 [admin] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-50x43.png [admin-width] => 50 [admin-height] => 43 [full] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [full-width] => 314 [full-height] => 268 [thumb_372] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [thumb_372-width] => 314 [thumb_372-height] => 268 [single_post_thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [single_post_thumbnail-width] => 314 [single_post_thumbnail-height] => 268 [thumb_32_32] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-32x27.png [thumb_32_32-width] => 32 [thumb_32_32-height] => 27 [thumb_48_48] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-48x41.png [thumb_48_48-width] => 48 [thumb_48_48-height] => 41 [thumb_64_64] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-64x55.png [thumb_64_64-width] => 64 [thumb_64_64-height] => 55 [thumb_128_128] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-128x109.png [thumb_128_128-width] => 128 [thumb_128_128-height] => 109 [thumb_122_67] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-79x67.png [thumb_122_67-width] => 79 [thumb_122_67-height] => 67 [thumb_153_153] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-153x131.png [thumb_153_153-width] => 153 [thumb_153_153-height] => 131 [thumb_130_163] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-130x111.png [thumb_130_163-width] => 130 [thumb_130_163-height] => 111 [thumb_232_177] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-207x177.png [thumb_232_177-width] => 207 [thumb_232_177-height] => 177 [thumb_272_325] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [thumb_272_325-width] => 314 [thumb_272_325-height] => 268 [thumb_271_346] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-271x231.png [thumb_271_346-width] => 271 [thumb_271_346-height] => 231 [thumb_271_271] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-271x231.png [thumb_271_271-width] => 271 [thumb_271_271-height] => 231 [thumb_272_172] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-202x172.png [thumb_272_172-width] => 202 [thumb_272_172-height] => 172 [thumb_372_461] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [thumb_372_461-width] => 314 [thumb_372_461-height] => 268 [thumb_372_500] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [thumb_372_500-width] => 314 [thumb_372_500-height] => 268 [thumb_385_270] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [thumb_385_270-width] => 314 [thumb_385_270-height] => 268 [thumb_416_241] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13-282x241.png [thumb_416_241-width] => 282 [thumb_416_241-height] => 241 [thumb_448_410] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [thumb_448_410-width] => 314 [thumb_448_410-height] => 268 [thumb_448_410_retina] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [thumb_448_410_retina-width] => 314 [thumb_448_410_retina-height] => 268 [thumb_574_382] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [thumb_574_382-width] => 314 [thumb_574_382-height] => 268 [thumb_776_395] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-13.png [thumb_776_395-width] => 314 [thumb_776_395-height] => 268 ) ) )

Results

 Treatment NameClick RateRelative DifferenceConfidence
C: Control 1.8%
T1: Treatment #1 0.80%-55.4% 100.0%

This experiment has a required sample size of 839 in order to be valid. Since the experiment had a total sample size of 1,407,300, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    55.4% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The key learning from this experiment is that using the term “Deep State” in Google ads instead of the more straightforward “Administrative State” led to a significant decrease in clicks. Contrary to the hypothesis, the more provocative term did not invoke stronger emotions or prompt more action from viewers.

In addition to click rate, the treatment also saw a 42.3% decrease in email acquisitions on the survey at 100% level of confidence, and an invalid 2.7% decrease in donor acquisition rate (6.82% level of confidence). The email acquisition results also support the same thought that the more provocative term did not invoke stronger emotions and action from viewers.

In future experimentation, it is important to carefully consider the language and terminology used in promotional materials. While provocative terms may sometimes generate interest, they can also backfire and deter potential participants. It is essential to test different messaging strategies to see what resonates best with the target audience before implementing them widely. Additionally, it is crucial to always track and analyze the results of experiments to draw meaningful conclusions and inform future decision-making.


Experiment Documented by NextAfter

Question about experiment #194779

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.