let us help
Subscribe
The Heritage Foundation

How leading a membership ask email with the problem for non-members affects click-through rate and donor conversion

Experiment ID: #169192

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Timeframe: 01/16/2024 - 01/16/2024

The Heritage Foundation is running thier annual Kickoff Campaign to acquire members. You can become a member by giving a gift of any size, one-time or recurring. This experiment was ran on the first email of the campaign for non-members to Heritage asking them to activate their membership. For this test, we wanted to see how non-members would react to leading the email with a question that poses the problem Heritage is trying to solve rather than leading the email with a direct ask to become a member. In doing so, we believe users will be more likely to click through to the donation page and activate their membership because the leading question allows them to relate to the issue and want to be apart of the change.

Research Question

We believe that leading an email with the problem will increase email click-through rate because it will give the reader more context as to why they should renew their membership with Heritage.

Design

C: Control
Array ( [0] => Array ( [ID] => 57117 [id] => 57117 [title] => Control.png [filename] => Control-116.png [filesize] => 135177 [url] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116.png [link] => https://www.nextafter.com/experiments/how-leading-a-membership-ask-email-with-the-problem-for-non-members-affects-click-through-rate-and-donor-conversion/attachment/control-png-6619/ [alt] => [author] => 0 [description] => [caption] => [name] => control-png-6619 [status] => inherit [uploaded_to] => 57077 [date] => 2024-01-31 16:39:09 [modified] => 2024-01-31 16:39:09 [menu_order] => 0 [mime_type] => image/png [type] => image [subtype] => png [icon] => https://nextafter-1a91a.kxcdn.com/wp-includes/images/media/default.png [width] => 1350 [height] => 808 [sizes] => Array ( [thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-150x150.png [thumbnail-width] => 150 [thumbnail-height] => 150 [medium] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-300x180.png [medium-width] => 300 [medium-height] => 180 [medium_large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-768x460.png [medium_large-width] => 640 [medium_large-height] => 383 [large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-1024x613.png [large-width] => 640 [large-height] => 383 [1536x1536] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116.png [1536x1536-width] => 1350 [1536x1536-height] => 808 [2048x2048] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116.png [2048x2048-width] => 1350 [2048x2048-height] => 808 [ab-block-post-grid-landscape] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-600x400.png [ab-block-post-grid-landscape-width] => 600 [ab-block-post-grid-landscape-height] => 400 [ab-block-post-grid-square] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-600x600.png [ab-block-post-grid-square-width] => 600 [ab-block-post-grid-square-height] => 600 [tp-image-grid] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116.png [tp-image-grid-width] => 700 [tp-image-grid-height] => 419 [post-thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-150x150.png [post-thumbnail-width] => 150 [post-thumbnail-height] => 150 [admin] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-50x30.png [admin-width] => 50 [admin-height] => 30 [full] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116.png [full-width] => 1350 [full-height] => 808 [thumb_372] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-372x223.png [thumb_372-width] => 372 [thumb_372-height] => 223 [single_post_thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-980x587.png [single_post_thumbnail-width] => 980 [single_post_thumbnail-height] => 587 [thumb_32_32] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-32x19.png [thumb_32_32-width] => 32 [thumb_32_32-height] => 19 [thumb_48_48] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-48x29.png [thumb_48_48-width] => 48 [thumb_48_48-height] => 29 [thumb_64_64] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-64x38.png [thumb_64_64-width] => 64 [thumb_64_64-height] => 38 [thumb_128_128] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-128x77.png [thumb_128_128-width] => 128 [thumb_128_128-height] => 77 [thumb_122_67] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-112x67.png [thumb_122_67-width] => 112 [thumb_122_67-height] => 67 [thumb_153_153] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-153x92.png [thumb_153_153-width] => 153 [thumb_153_153-height] => 92 [thumb_130_163] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-130x78.png [thumb_130_163-width] => 130 [thumb_130_163-height] => 78 [thumb_232_177] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-232x139.png [thumb_232_177-width] => 232 [thumb_232_177-height] => 139 [thumb_272_325] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-544x326.png [thumb_272_325-width] => 544 [thumb_272_325-height] => 326 [thumb_271_346] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-271x162.png [thumb_271_346-width] => 271 [thumb_271_346-height] => 162 [thumb_271_271] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-271x162.png [thumb_271_271-width] => 271 [thumb_271_271-height] => 162 [thumb_272_172] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-272x163.png [thumb_272_172-width] => 272 [thumb_272_172-height] => 163 [thumb_372_461] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-372x223.png [thumb_372_461-width] => 372 [thumb_372_461-height] => 223 [thumb_372_500] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-372x223.png [thumb_372_500-width] => 372 [thumb_372_500-height] => 223 [thumb_385_270] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-385x230.png [thumb_385_270-width] => 385 [thumb_385_270-height] => 230 [thumb_416_241] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-403x241.png [thumb_416_241-width] => 403 [thumb_416_241-height] => 241 [thumb_448_410] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-448x268.png [thumb_448_410-width] => 448 [thumb_448_410-height] => 268 [thumb_448_410_retina] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-896x536.png [thumb_448_410_retina-width] => 896 [thumb_448_410_retina-height] => 536 [thumb_574_382] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-574x344.png [thumb_574_382-width] => 574 [thumb_574_382-height] => 344 [thumb_776_395] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-660x395.png [thumb_776_395-width] => 660 [thumb_776_395-height] => 395 ) ) [1] => Array ( [ID] => 57118 [id] => 57118 [title] => Treatment_#1.png [filename] => Treatment_1-67.png [filesize] => 150924 [url] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67.png [link] => https://www.nextafter.com/experiments/how-leading-a-membership-ask-email-with-the-problem-for-non-members-affects-click-through-rate-and-donor-conversion/attachment/treatment_1-png-2116/ [alt] => [author] => 0 [description] => [caption] => [name] => treatment_1-png-2116 [status] => inherit [uploaded_to] => 57077 [date] => 2024-01-31 16:39:16 [modified] => 2024-01-31 16:39:16 [menu_order] => 0 [mime_type] => image/png [type] => image [subtype] => png [icon] => https://nextafter-1a91a.kxcdn.com/wp-includes/images/media/default.png [width] => 1348 [height] => 866 [sizes] => Array ( [thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-150x150.png [thumbnail-width] => 150 [thumbnail-height] => 150 [medium] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-300x193.png [medium-width] => 300 [medium-height] => 193 [medium_large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-768x493.png [medium_large-width] => 640 [medium_large-height] => 411 [large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-1024x658.png [large-width] => 640 [large-height] => 411 [1536x1536] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67.png [1536x1536-width] => 1348 [1536x1536-height] => 866 [2048x2048] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67.png [2048x2048-width] => 1348 [2048x2048-height] => 866 [ab-block-post-grid-landscape] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-600x400.png [ab-block-post-grid-landscape-width] => 600 [ab-block-post-grid-landscape-height] => 400 [ab-block-post-grid-square] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-600x600.png [ab-block-post-grid-square-width] => 600 [ab-block-post-grid-square-height] => 600 [tp-image-grid] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67.png [tp-image-grid-width] => 700 [tp-image-grid-height] => 450 [post-thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-150x150.png [post-thumbnail-width] => 150 [post-thumbnail-height] => 150 [admin] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-50x32.png [admin-width] => 50 [admin-height] => 32 [full] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67.png [full-width] => 1348 [full-height] => 866 [thumb_372] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-372x239.png [thumb_372-width] => 372 [thumb_372-height] => 239 [single_post_thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-973x625.png [single_post_thumbnail-width] => 973 [single_post_thumbnail-height] => 625 [thumb_32_32] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-32x21.png [thumb_32_32-width] => 32 [thumb_32_32-height] => 21 [thumb_48_48] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-48x31.png [thumb_48_48-width] => 48 [thumb_48_48-height] => 31 [thumb_64_64] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-64x41.png [thumb_64_64-width] => 64 [thumb_64_64-height] => 41 [thumb_128_128] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-128x82.png [thumb_128_128-width] => 128 [thumb_128_128-height] => 82 [thumb_122_67] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-104x67.png [thumb_122_67-width] => 104 [thumb_122_67-height] => 67 [thumb_153_153] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-153x98.png [thumb_153_153-width] => 153 [thumb_153_153-height] => 98 [thumb_130_163] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-130x84.png [thumb_130_163-width] => 130 [thumb_130_163-height] => 84 [thumb_232_177] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-232x149.png [thumb_232_177-width] => 232 [thumb_232_177-height] => 149 [thumb_272_325] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-544x349.png [thumb_272_325-width] => 544 [thumb_272_325-height] => 349 [thumb_271_346] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-271x174.png [thumb_271_346-width] => 271 [thumb_271_346-height] => 174 [thumb_271_271] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-271x174.png [thumb_271_271-width] => 271 [thumb_271_271-height] => 174 [thumb_272_172] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-268x172.png [thumb_272_172-width] => 268 [thumb_272_172-height] => 172 [thumb_372_461] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-372x239.png [thumb_372_461-width] => 372 [thumb_372_461-height] => 239 [thumb_372_500] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-372x239.png [thumb_372_500-width] => 372 [thumb_372_500-height] => 239 [thumb_385_270] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-385x247.png [thumb_385_270-width] => 385 [thumb_385_270-height] => 247 [thumb_416_241] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-375x241.png [thumb_416_241-width] => 375 [thumb_416_241-height] => 241 [thumb_448_410] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-448x288.png [thumb_448_410-width] => 448 [thumb_448_410-height] => 288 [thumb_448_410_retina] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-896x576.png [thumb_448_410_retina-width] => 896 [thumb_448_410_retina-height] => 576 [thumb_574_382] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-574x369.png [thumb_574_382-width] => 574 [thumb_574_382-height] => 369 [thumb_776_395] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-615x395.png [thumb_776_395-width] => 615 [thumb_776_395-height] => 395 ) ) )
T1: Treatment #1
Array ( [0] => Array ( [ID] => 57117 [id] => 57117 [title] => Control.png [filename] => Control-116.png [filesize] => 135177 [url] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116.png [link] => https://www.nextafter.com/experiments/how-leading-a-membership-ask-email-with-the-problem-for-non-members-affects-click-through-rate-and-donor-conversion/attachment/control-png-6619/ [alt] => [author] => 0 [description] => [caption] => [name] => control-png-6619 [status] => inherit [uploaded_to] => 57077 [date] => 2024-01-31 16:39:09 [modified] => 2024-01-31 16:39:09 [menu_order] => 0 [mime_type] => image/png [type] => image [subtype] => png [icon] => https://nextafter-1a91a.kxcdn.com/wp-includes/images/media/default.png [width] => 1350 [height] => 808 [sizes] => Array ( [thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-150x150.png [thumbnail-width] => 150 [thumbnail-height] => 150 [medium] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-300x180.png [medium-width] => 300 [medium-height] => 180 [medium_large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-768x460.png [medium_large-width] => 640 [medium_large-height] => 383 [large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-1024x613.png [large-width] => 640 [large-height] => 383 [1536x1536] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116.png [1536x1536-width] => 1350 [1536x1536-height] => 808 [2048x2048] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116.png [2048x2048-width] => 1350 [2048x2048-height] => 808 [ab-block-post-grid-landscape] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-600x400.png [ab-block-post-grid-landscape-width] => 600 [ab-block-post-grid-landscape-height] => 400 [ab-block-post-grid-square] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-600x600.png [ab-block-post-grid-square-width] => 600 [ab-block-post-grid-square-height] => 600 [tp-image-grid] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116.png [tp-image-grid-width] => 700 [tp-image-grid-height] => 419 [post-thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-150x150.png [post-thumbnail-width] => 150 [post-thumbnail-height] => 150 [admin] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-50x30.png [admin-width] => 50 [admin-height] => 30 [full] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116.png [full-width] => 1350 [full-height] => 808 [thumb_372] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-372x223.png [thumb_372-width] => 372 [thumb_372-height] => 223 [single_post_thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-980x587.png [single_post_thumbnail-width] => 980 [single_post_thumbnail-height] => 587 [thumb_32_32] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-32x19.png [thumb_32_32-width] => 32 [thumb_32_32-height] => 19 [thumb_48_48] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-48x29.png [thumb_48_48-width] => 48 [thumb_48_48-height] => 29 [thumb_64_64] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-64x38.png [thumb_64_64-width] => 64 [thumb_64_64-height] => 38 [thumb_128_128] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-128x77.png [thumb_128_128-width] => 128 [thumb_128_128-height] => 77 [thumb_122_67] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-112x67.png [thumb_122_67-width] => 112 [thumb_122_67-height] => 67 [thumb_153_153] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-153x92.png [thumb_153_153-width] => 153 [thumb_153_153-height] => 92 [thumb_130_163] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-130x78.png [thumb_130_163-width] => 130 [thumb_130_163-height] => 78 [thumb_232_177] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-232x139.png [thumb_232_177-width] => 232 [thumb_232_177-height] => 139 [thumb_272_325] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-544x326.png [thumb_272_325-width] => 544 [thumb_272_325-height] => 326 [thumb_271_346] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-271x162.png [thumb_271_346-width] => 271 [thumb_271_346-height] => 162 [thumb_271_271] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-271x162.png [thumb_271_271-width] => 271 [thumb_271_271-height] => 162 [thumb_272_172] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-272x163.png [thumb_272_172-width] => 272 [thumb_272_172-height] => 163 [thumb_372_461] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-372x223.png [thumb_372_461-width] => 372 [thumb_372_461-height] => 223 [thumb_372_500] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-372x223.png [thumb_372_500-width] => 372 [thumb_372_500-height] => 223 [thumb_385_270] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-385x230.png [thumb_385_270-width] => 385 [thumb_385_270-height] => 230 [thumb_416_241] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-403x241.png [thumb_416_241-width] => 403 [thumb_416_241-height] => 241 [thumb_448_410] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-448x268.png [thumb_448_410-width] => 448 [thumb_448_410-height] => 268 [thumb_448_410_retina] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-896x536.png [thumb_448_410_retina-width] => 896 [thumb_448_410_retina-height] => 536 [thumb_574_382] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-574x344.png [thumb_574_382-width] => 574 [thumb_574_382-height] => 344 [thumb_776_395] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-116-660x395.png [thumb_776_395-width] => 660 [thumb_776_395-height] => 395 ) ) [1] => Array ( [ID] => 57118 [id] => 57118 [title] => Treatment_#1.png [filename] => Treatment_1-67.png [filesize] => 150924 [url] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67.png [link] => https://www.nextafter.com/experiments/how-leading-a-membership-ask-email-with-the-problem-for-non-members-affects-click-through-rate-and-donor-conversion/attachment/treatment_1-png-2116/ [alt] => [author] => 0 [description] => [caption] => [name] => treatment_1-png-2116 [status] => inherit [uploaded_to] => 57077 [date] => 2024-01-31 16:39:16 [modified] => 2024-01-31 16:39:16 [menu_order] => 0 [mime_type] => image/png [type] => image [subtype] => png [icon] => https://nextafter-1a91a.kxcdn.com/wp-includes/images/media/default.png [width] => 1348 [height] => 866 [sizes] => Array ( [thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-150x150.png [thumbnail-width] => 150 [thumbnail-height] => 150 [medium] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-300x193.png [medium-width] => 300 [medium-height] => 193 [medium_large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-768x493.png [medium_large-width] => 640 [medium_large-height] => 411 [large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-1024x658.png [large-width] => 640 [large-height] => 411 [1536x1536] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67.png [1536x1536-width] => 1348 [1536x1536-height] => 866 [2048x2048] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67.png [2048x2048-width] => 1348 [2048x2048-height] => 866 [ab-block-post-grid-landscape] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-600x400.png [ab-block-post-grid-landscape-width] => 600 [ab-block-post-grid-landscape-height] => 400 [ab-block-post-grid-square] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-600x600.png [ab-block-post-grid-square-width] => 600 [ab-block-post-grid-square-height] => 600 [tp-image-grid] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67.png [tp-image-grid-width] => 700 [tp-image-grid-height] => 450 [post-thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-150x150.png [post-thumbnail-width] => 150 [post-thumbnail-height] => 150 [admin] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-50x32.png [admin-width] => 50 [admin-height] => 32 [full] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67.png [full-width] => 1348 [full-height] => 866 [thumb_372] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-372x239.png [thumb_372-width] => 372 [thumb_372-height] => 239 [single_post_thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-973x625.png [single_post_thumbnail-width] => 973 [single_post_thumbnail-height] => 625 [thumb_32_32] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-32x21.png [thumb_32_32-width] => 32 [thumb_32_32-height] => 21 [thumb_48_48] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-48x31.png [thumb_48_48-width] => 48 [thumb_48_48-height] => 31 [thumb_64_64] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-64x41.png [thumb_64_64-width] => 64 [thumb_64_64-height] => 41 [thumb_128_128] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-128x82.png [thumb_128_128-width] => 128 [thumb_128_128-height] => 82 [thumb_122_67] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-104x67.png [thumb_122_67-width] => 104 [thumb_122_67-height] => 67 [thumb_153_153] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-153x98.png [thumb_153_153-width] => 153 [thumb_153_153-height] => 98 [thumb_130_163] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-130x84.png [thumb_130_163-width] => 130 [thumb_130_163-height] => 84 [thumb_232_177] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-232x149.png [thumb_232_177-width] => 232 [thumb_232_177-height] => 149 [thumb_272_325] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-544x349.png [thumb_272_325-width] => 544 [thumb_272_325-height] => 349 [thumb_271_346] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-271x174.png [thumb_271_346-width] => 271 [thumb_271_346-height] => 174 [thumb_271_271] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-271x174.png [thumb_271_271-width] => 271 [thumb_271_271-height] => 174 [thumb_272_172] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-268x172.png [thumb_272_172-width] => 268 [thumb_272_172-height] => 172 [thumb_372_461] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-372x239.png [thumb_372_461-width] => 372 [thumb_372_461-height] => 239 [thumb_372_500] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-372x239.png [thumb_372_500-width] => 372 [thumb_372_500-height] => 239 [thumb_385_270] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-385x247.png [thumb_385_270-width] => 385 [thumb_385_270-height] => 247 [thumb_416_241] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-375x241.png [thumb_416_241-width] => 375 [thumb_416_241-height] => 241 [thumb_448_410] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-448x288.png [thumb_448_410-width] => 448 [thumb_448_410-height] => 288 [thumb_448_410_retina] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-896x576.png [thumb_448_410_retina-width] => 896 [thumb_448_410_retina-height] => 576 [thumb_574_382] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-574x369.png [thumb_574_382-width] => 574 [thumb_574_382-height] => 369 [thumb_776_395] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Treatment_1-67-615x395.png [thumb_776_395-width] => 615 [thumb_776_395-height] => 395 ) ) )

Results

 Treatment NameClick RateRelative DifferenceConfidence
C: Control 0.61%
T1: Treatment #1 0.37%-39.6% 100.0%

This experiment has a required sample size of 6,448 in order to be valid. Since the experiment had a total sample size of 628,905, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    39.6% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The key learning from this experiment is that leading an email with a problem statement did not increase email click-through rates for new members. In fact, the treatment group experienced a 39% decrease in clicks compared to the control group. In addition, the treatment group saw a 61% decrease in donations with a 99.9% level of confidence.

Based on this result, it is clear that the approach of posing a problem as the lead in the email is not effective in motivating non-members to click through and make a donation. In future experimentation, it would be advisable to consider alternative approaches to engage non-members and encourage them to become members. This could involve highlighting the benefits of membership, showcasing successful outcomes of past campaigns, or offering exclusive content or promotions for members. Additionally, it would be beneficial to conduct further research to understand the reasons behind the negative response to the problem statement approach, such as through qualitative interviews or surveys.

Experiment Documented by NextAfter

Question about experiment #169192

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.