let us help
Subscribe
The Heritage Foundation

How testing a new offer in a “Dear Reader” ad affected clicks and donations

Experiment ID: #35722

The Heritage Foundation

Experiment Summary

Timeframe: 09/11/2020 - 10/24/2020

As we prepared to launch a new offer to the housefile and in acquisition, we were faced with a dilemma. The new eBook offer was relevant and time-sensitive, but we didn’t want to kick out an offer that was already performing strongly and miss out on additional conversions by losing a month of run time. So instead of replacing the current Dear Reader eBook offer outright, we decided to make the new offer compete against the old offer for the spot.

Research Question

We believe that offering a relevant, time-sensitive resource for website visitors will achieve increase clicks and donations.

Design

C: Democratic Socialism eBook
Array ( [0] => Array ( [ID] => 37722 [id] => 37722 [title] => Democratic_Socialism_eBook.png [filename] => Democratic_Socialism_eBook.png [filesize] => 181217 [url] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [link] => https://www.nextafter.com/experiments/how-testing-a-new-offer-in-a-dear-reader-ad-affected-clicks-and-donations/attachment/democratic_socialism_ebook-png/ [alt] => [author] => 0 [description] => [caption] => [name] => democratic_socialism_ebook-png [status] => inherit [uploaded_to] => 37721 [date] => 2022-10-18 16:10:07 [modified] => 2022-10-18 16:10:07 [menu_order] => 0 [mime_type] => image/png [type] => image [subtype] => png [icon] => https://www.nextafter.com/wp-includes/images/media/default.png [width] => 626 [height] => 818 [sizes] => Array ( [thumbnail] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-150x150.png [thumbnail-width] => 150 [thumbnail-height] => 150 [medium] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-230x300.png [medium-width] => 230 [medium-height] => 300 [medium_large] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [medium_large-width] => 626 [medium_large-height] => 818 [large] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [large-width] => 626 [large-height] => 818 [1536x1536] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [1536x1536-width] => 626 [1536x1536-height] => 818 [2048x2048] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [2048x2048-width] => 626 [2048x2048-height] => 818 [tp-image-grid] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [tp-image-grid-width] => 536 [tp-image-grid-height] => 700 [post-thumbnail] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-150x150.png [post-thumbnail-width] => 150 [post-thumbnail-height] => 150 [admin] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-38x50.png [admin-width] => 38 [admin-height] => 50 [full] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [full-width] => 626 [full-height] => 818 [thumb_372] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-372x486.png [thumb_372-width] => 372 [thumb_372-height] => 486 [single_post_thumbnail] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-478x625.png [single_post_thumbnail-width] => 478 [single_post_thumbnail-height] => 625 [thumb_32_32] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-24x32.png [thumb_32_32-width] => 24 [thumb_32_32-height] => 32 [thumb_48_48] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-37x48.png [thumb_48_48-width] => 37 [thumb_48_48-height] => 48 [thumb_64_64] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-49x64.png [thumb_64_64-width] => 49 [thumb_64_64-height] => 64 [thumb_128_128] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-98x128.png [thumb_128_128-width] => 98 [thumb_128_128-height] => 128 [thumb_122_67] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-51x67.png [thumb_122_67-width] => 51 [thumb_122_67-height] => 67 [thumb_153_153] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-117x153.png [thumb_153_153-width] => 117 [thumb_153_153-height] => 153 [thumb_130_163] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-125x163.png [thumb_130_163-width] => 125 [thumb_130_163-height] => 163 [thumb_232_177] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-135x177.png [thumb_232_177-width] => 135 [thumb_232_177-height] => 177 [thumb_272_325] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-497x650.png [thumb_272_325-width] => 497 [thumb_272_325-height] => 650 [thumb_271_346] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-265x346.png [thumb_271_346-width] => 265 [thumb_271_346-height] => 346 [thumb_271_271] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-207x271.png [thumb_271_271-width] => 207 [thumb_271_271-height] => 271 [thumb_272_172] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-132x172.png [thumb_272_172-width] => 132 [thumb_272_172-height] => 172 [thumb_372_461] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-353x461.png [thumb_372_461-width] => 353 [thumb_372_461-height] => 461 [thumb_372_500] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-372x486.png [thumb_372_500-width] => 372 [thumb_372_500-height] => 486 [thumb_385_270] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-207x270.png [thumb_385_270-width] => 207 [thumb_385_270-height] => 270 [thumb_416_241] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-184x241.png [thumb_416_241-width] => 184 [thumb_416_241-height] => 241 [thumb_448_410] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-314x410.png [thumb_448_410-width] => 314 [thumb_448_410-height] => 410 [thumb_448_410_retina] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [thumb_448_410_retina-width] => 626 [thumb_448_410_retina-height] => 818 [thumb_574_382] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-292x382.png [thumb_574_382-width] => 292 [thumb_574_382-height] => 382 [thumb_776_395] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-302x395.png [thumb_776_395-width] => 302 [thumb_776_395-height] => 395 [gform-image-choice-sm] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [gform-image-choice-sm-width] => 230 [gform-image-choice-sm-height] => 300 [gform-image-choice-md] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [gform-image-choice-md-width] => 306 [gform-image-choice-md-height] => 400 [gform-image-choice-lg] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [gform-image-choice-lg-width] => 459 [gform-image-choice-lg-height] => 600 ) ) [1] => Array ( [ID] => 37727 [id] => 37727 [title] => MFL_Election_Integrity.png [filename] => MFL_Election_Integrity.png [filesize] => 184432 [url] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [link] => https://www.nextafter.com/experiments/how-testing-a-new-offer-in-a-dear-reader-ad-affected-clicks-and-donations/attachment/mfl_election_integrity-png-2/ [alt] => [author] => 0 [description] => [caption] => [name] => mfl_election_integrity-png-2 [status] => inherit [uploaded_to] => 37721 [date] => 2022-10-18 16:10:46 [modified] => 2022-10-18 16:10:46 [menu_order] => 0 [mime_type] => image/png [type] => image [subtype] => png [icon] => https://www.nextafter.com/wp-includes/images/media/default.png [width] => 612 [height] => 600 [sizes] => Array ( [thumbnail] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-150x150.png [thumbnail-width] => 150 [thumbnail-height] => 150 [medium] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-300x294.png [medium-width] => 300 [medium-height] => 294 [medium_large] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [medium_large-width] => 612 [medium_large-height] => 600 [large] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [large-width] => 612 [large-height] => 600 [1536x1536] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [1536x1536-width] => 612 [1536x1536-height] => 600 [2048x2048] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [2048x2048-width] => 612 [2048x2048-height] => 600 [tp-image-grid] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [tp-image-grid-width] => 612 [tp-image-grid-height] => 600 [post-thumbnail] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-150x150.png [post-thumbnail-width] => 150 [post-thumbnail-height] => 150 [admin] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-50x50.png [admin-width] => 50 [admin-height] => 50 [full] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [full-width] => 612 [full-height] => 600 [thumb_372] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-372x365.png [thumb_372-width] => 372 [thumb_372-height] => 365 [single_post_thumbnail] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [single_post_thumbnail-width] => 612 [single_post_thumbnail-height] => 600 [thumb_32_32] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-32x32.png [thumb_32_32-width] => 32 [thumb_32_32-height] => 32 [thumb_48_48] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-48x48.png [thumb_48_48-width] => 48 [thumb_48_48-height] => 48 [thumb_64_64] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-64x64.png [thumb_64_64-width] => 64 [thumb_64_64-height] => 64 [thumb_128_128] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-128x125.png [thumb_128_128-width] => 128 [thumb_128_128-height] => 125 [thumb_122_67] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-68x67.png [thumb_122_67-width] => 68 [thumb_122_67-height] => 67 [thumb_153_153] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-153x150.png [thumb_153_153-width] => 153 [thumb_153_153-height] => 150 [thumb_130_163] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-130x127.png [thumb_130_163-width] => 130 [thumb_130_163-height] => 127 [thumb_232_177] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-181x177.png [thumb_232_177-width] => 181 [thumb_232_177-height] => 177 [thumb_272_325] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-544x533.png [thumb_272_325-width] => 544 [thumb_272_325-height] => 533 [thumb_271_346] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-271x266.png [thumb_271_346-width] => 271 [thumb_271_346-height] => 266 [thumb_271_271] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-271x266.png [thumb_271_271-width] => 271 [thumb_271_271-height] => 266 [thumb_272_172] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-175x172.png [thumb_272_172-width] => 175 [thumb_272_172-height] => 172 [thumb_372_461] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-372x365.png [thumb_372_461-width] => 372 [thumb_372_461-height] => 365 [thumb_372_500] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-372x365.png [thumb_372_500-width] => 372 [thumb_372_500-height] => 365 [thumb_385_270] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-275x270.png [thumb_385_270-width] => 275 [thumb_385_270-height] => 270 [thumb_416_241] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-246x241.png [thumb_416_241-width] => 246 [thumb_416_241-height] => 241 [thumb_448_410] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-418x410.png [thumb_448_410-width] => 418 [thumb_448_410-height] => 410 [thumb_448_410_retina] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [thumb_448_410_retina-width] => 612 [thumb_448_410_retina-height] => 600 [thumb_574_382] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-390x382.png [thumb_574_382-width] => 390 [thumb_574_382-height] => 382 [thumb_776_395] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-403x395.png [thumb_776_395-width] => 403 [thumb_776_395-height] => 395 [gform-image-choice-sm] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [gform-image-choice-sm-width] => 300 [gform-image-choice-sm-height] => 294 [gform-image-choice-md] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [gform-image-choice-md-width] => 400 [gform-image-choice-md-height] => 392 [gform-image-choice-lg] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [gform-image-choice-lg-width] => 600 [gform-image-choice-lg-height] => 588 ) ) )
T1: MFL Election Integrity
Array ( [0] => Array ( [ID] => 37722 [id] => 37722 [title] => Democratic_Socialism_eBook.png [filename] => Democratic_Socialism_eBook.png [filesize] => 181217 [url] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [link] => https://www.nextafter.com/experiments/how-testing-a-new-offer-in-a-dear-reader-ad-affected-clicks-and-donations/attachment/democratic_socialism_ebook-png/ [alt] => [author] => 0 [description] => [caption] => [name] => democratic_socialism_ebook-png [status] => inherit [uploaded_to] => 37721 [date] => 2022-10-18 16:10:07 [modified] => 2022-10-18 16:10:07 [menu_order] => 0 [mime_type] => image/png [type] => image [subtype] => png [icon] => https://www.nextafter.com/wp-includes/images/media/default.png [width] => 626 [height] => 818 [sizes] => Array ( [thumbnail] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-150x150.png [thumbnail-width] => 150 [thumbnail-height] => 150 [medium] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-230x300.png [medium-width] => 230 [medium-height] => 300 [medium_large] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [medium_large-width] => 626 [medium_large-height] => 818 [large] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [large-width] => 626 [large-height] => 818 [1536x1536] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [1536x1536-width] => 626 [1536x1536-height] => 818 [2048x2048] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [2048x2048-width] => 626 [2048x2048-height] => 818 [tp-image-grid] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [tp-image-grid-width] => 536 [tp-image-grid-height] => 700 [post-thumbnail] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-150x150.png [post-thumbnail-width] => 150 [post-thumbnail-height] => 150 [admin] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-38x50.png [admin-width] => 38 [admin-height] => 50 [full] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [full-width] => 626 [full-height] => 818 [thumb_372] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-372x486.png [thumb_372-width] => 372 [thumb_372-height] => 486 [single_post_thumbnail] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-478x625.png [single_post_thumbnail-width] => 478 [single_post_thumbnail-height] => 625 [thumb_32_32] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-24x32.png [thumb_32_32-width] => 24 [thumb_32_32-height] => 32 [thumb_48_48] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-37x48.png [thumb_48_48-width] => 37 [thumb_48_48-height] => 48 [thumb_64_64] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-49x64.png [thumb_64_64-width] => 49 [thumb_64_64-height] => 64 [thumb_128_128] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-98x128.png [thumb_128_128-width] => 98 [thumb_128_128-height] => 128 [thumb_122_67] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-51x67.png [thumb_122_67-width] => 51 [thumb_122_67-height] => 67 [thumb_153_153] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-117x153.png [thumb_153_153-width] => 117 [thumb_153_153-height] => 153 [thumb_130_163] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-125x163.png [thumb_130_163-width] => 125 [thumb_130_163-height] => 163 [thumb_232_177] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-135x177.png [thumb_232_177-width] => 135 [thumb_232_177-height] => 177 [thumb_272_325] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-497x650.png [thumb_272_325-width] => 497 [thumb_272_325-height] => 650 [thumb_271_346] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-265x346.png [thumb_271_346-width] => 265 [thumb_271_346-height] => 346 [thumb_271_271] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-207x271.png [thumb_271_271-width] => 207 [thumb_271_271-height] => 271 [thumb_272_172] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-132x172.png [thumb_272_172-width] => 132 [thumb_272_172-height] => 172 [thumb_372_461] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-353x461.png [thumb_372_461-width] => 353 [thumb_372_461-height] => 461 [thumb_372_500] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-372x486.png [thumb_372_500-width] => 372 [thumb_372_500-height] => 486 [thumb_385_270] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-207x270.png [thumb_385_270-width] => 207 [thumb_385_270-height] => 270 [thumb_416_241] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-184x241.png [thumb_416_241-width] => 184 [thumb_416_241-height] => 241 [thumb_448_410] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-314x410.png [thumb_448_410-width] => 314 [thumb_448_410-height] => 410 [thumb_448_410_retina] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [thumb_448_410_retina-width] => 626 [thumb_448_410_retina-height] => 818 [thumb_574_382] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-292x382.png [thumb_574_382-width] => 292 [thumb_574_382-height] => 382 [thumb_776_395] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook-302x395.png [thumb_776_395-width] => 302 [thumb_776_395-height] => 395 [gform-image-choice-sm] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [gform-image-choice-sm-width] => 230 [gform-image-choice-sm-height] => 300 [gform-image-choice-md] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [gform-image-choice-md-width] => 306 [gform-image-choice-md-height] => 400 [gform-image-choice-lg] => https://www.nextafter.com/wp-content/uploads/Democratic_Socialism_eBook.png [gform-image-choice-lg-width] => 459 [gform-image-choice-lg-height] => 600 ) ) [1] => Array ( [ID] => 37727 [id] => 37727 [title] => MFL_Election_Integrity.png [filename] => MFL_Election_Integrity.png [filesize] => 184432 [url] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [link] => https://www.nextafter.com/experiments/how-testing-a-new-offer-in-a-dear-reader-ad-affected-clicks-and-donations/attachment/mfl_election_integrity-png-2/ [alt] => [author] => 0 [description] => [caption] => [name] => mfl_election_integrity-png-2 [status] => inherit [uploaded_to] => 37721 [date] => 2022-10-18 16:10:46 [modified] => 2022-10-18 16:10:46 [menu_order] => 0 [mime_type] => image/png [type] => image [subtype] => png [icon] => https://www.nextafter.com/wp-includes/images/media/default.png [width] => 612 [height] => 600 [sizes] => Array ( [thumbnail] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-150x150.png [thumbnail-width] => 150 [thumbnail-height] => 150 [medium] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-300x294.png [medium-width] => 300 [medium-height] => 294 [medium_large] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [medium_large-width] => 612 [medium_large-height] => 600 [large] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [large-width] => 612 [large-height] => 600 [1536x1536] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [1536x1536-width] => 612 [1536x1536-height] => 600 [2048x2048] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [2048x2048-width] => 612 [2048x2048-height] => 600 [tp-image-grid] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [tp-image-grid-width] => 612 [tp-image-grid-height] => 600 [post-thumbnail] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-150x150.png [post-thumbnail-width] => 150 [post-thumbnail-height] => 150 [admin] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-50x50.png [admin-width] => 50 [admin-height] => 50 [full] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [full-width] => 612 [full-height] => 600 [thumb_372] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-372x365.png [thumb_372-width] => 372 [thumb_372-height] => 365 [single_post_thumbnail] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [single_post_thumbnail-width] => 612 [single_post_thumbnail-height] => 600 [thumb_32_32] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-32x32.png [thumb_32_32-width] => 32 [thumb_32_32-height] => 32 [thumb_48_48] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-48x48.png [thumb_48_48-width] => 48 [thumb_48_48-height] => 48 [thumb_64_64] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-64x64.png [thumb_64_64-width] => 64 [thumb_64_64-height] => 64 [thumb_128_128] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-128x125.png [thumb_128_128-width] => 128 [thumb_128_128-height] => 125 [thumb_122_67] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-68x67.png [thumb_122_67-width] => 68 [thumb_122_67-height] => 67 [thumb_153_153] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-153x150.png [thumb_153_153-width] => 153 [thumb_153_153-height] => 150 [thumb_130_163] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-130x127.png [thumb_130_163-width] => 130 [thumb_130_163-height] => 127 [thumb_232_177] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-181x177.png [thumb_232_177-width] => 181 [thumb_232_177-height] => 177 [thumb_272_325] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-544x533.png [thumb_272_325-width] => 544 [thumb_272_325-height] => 533 [thumb_271_346] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-271x266.png [thumb_271_346-width] => 271 [thumb_271_346-height] => 266 [thumb_271_271] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-271x266.png [thumb_271_271-width] => 271 [thumb_271_271-height] => 266 [thumb_272_172] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-175x172.png [thumb_272_172-width] => 175 [thumb_272_172-height] => 172 [thumb_372_461] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-372x365.png [thumb_372_461-width] => 372 [thumb_372_461-height] => 365 [thumb_372_500] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-372x365.png [thumb_372_500-width] => 372 [thumb_372_500-height] => 365 [thumb_385_270] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-275x270.png [thumb_385_270-width] => 275 [thumb_385_270-height] => 270 [thumb_416_241] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-246x241.png [thumb_416_241-width] => 246 [thumb_416_241-height] => 241 [thumb_448_410] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-418x410.png [thumb_448_410-width] => 418 [thumb_448_410-height] => 410 [thumb_448_410_retina] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [thumb_448_410_retina-width] => 612 [thumb_448_410_retina-height] => 600 [thumb_574_382] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-390x382.png [thumb_574_382-width] => 390 [thumb_574_382-height] => 382 [thumb_776_395] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity-403x395.png [thumb_776_395-width] => 403 [thumb_776_395-height] => 395 [gform-image-choice-sm] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [gform-image-choice-sm-width] => 300 [gform-image-choice-sm-height] => 294 [gform-image-choice-md] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [gform-image-choice-md-width] => 400 [gform-image-choice-md-height] => 392 [gform-image-choice-lg] => https://www.nextafter.com/wp-content/uploads/MFL_Election_Integrity.png [gform-image-choice-lg-width] => 600 [gform-image-choice-lg-height] => 588 ) ) )

Results

  Treatment Name Conv. Rate Relative Difference Confidence
C: Democratic Socialism eBook 1.8%
T1: MFL Election Integrity 1.2% -32.8% 100.0%

This experiment has a required sample size of 3,297 in order to be valid. Since the experiment had a total sample size of 1,271,182, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 32.8% decrease in conversion rate
× 0% increase in average gift

Key Learnings

What we found is that the new offer had lower performance than the current offer across the board. We saw a 32.8% decrease in conversion rate for the new eBook offer, which you can see broken down by segment below, resulting in fewer engagements in both clicks and donations. These results helped us reach a conclusion to keep the existing offer in place rather than rolling out the new eBook offer in this high-traffic Dear Reader spot. New is not always better! For future new offers, we will plan to run additional A/B tests to determine which offer will lead to maximum results instead of giving away valuable real estate to an offer with unknown performance and risk losing potential revenue and have to make up for lost time later. This experimental approach with initially rolling out to only 50% of site visitors will help us focus less on if the offer is fresh and instead care more about if the offer works. Observations of the impact on other visitor segments in the experiment:

  • Direct Traffic visitors had a 52.4% decrease in cta click with a 100% level of confidence.
  • Email visitors had a 29.7% decrease in cta click with a 100% level of confidence.
  • Organic Search visitors had a 61.9% decrease in cta click with a 100% level of confidence.
  • Social visitors had a 49.1% decrease in cta click with a 100% level of confidence.
  • All Traffic visitors had a 32.8% decrease in cta click with a 100% level of confidence.
  • Desktop visitors had a 23.0% decrease in cta click with a 100% level of confidence.
  • Mobile visitors had a 40.4% decrease in cta click with a 100% level of confidence.
  • Tablet visitors had a 34.5% decrease in cta click with a 100% level of confidence.
  • New visitors had a 46.6% decrease in cta click with a 100% level of confidence.

Experiment Documented by Rebekah Josefy
Rebekah Josefy is an Optimization Director at NextAfter.

Question about experiment #35722

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.