How using an iPhone Notes App style of Facebook ad creative impact clicks
Experiment ID: #189262
Buckner International
Experiment Summary
Timeframe: 06/24/2024 - 07/12/2024
Buckner International is a Christian nonprofit committed to serving children and families, with their Shoes for Orphan Souls (SOS) campaign being a significant donor acquisition effort.
In this experiment, we compared the effectiveness of an iPhone notes app style ad to an organic-style text ad. The iPhone notes app style ad mimics the look of a personal notes application, featuring a familiar and engaging format. The organic-style text ad has a straightforward design with just text in a plain background. Our hypothesis is that the Notes App ads, with their authentic and eye-catching design, will significantly boost clicks and email signups. This is due to their unique appearance, which we believe will resonate more with our target audience on Facebook.
We set up two Advantage+ campaigns on Facebook with equal budgets for each ad style, and the test ran for 18 days. Both ad creatives have the same call-to-action: donate $15 to provide shoes but with slight differences in language.
Research Question
Our hypothesis is that the Notes App ads, with their authentic and eye-catching design, will significantly boost clicks and email signups. This is due to their unique appearance, which we believe will resonate more with our target audience on Facebook.
Design
C: Control
Array
(
[0] => Array
(
[ID] => 64736
[id] => 64736
[title] => Control.png
[filename] => Control-800.png
[filesize] => 187041
[url] => https://www.nextafter.com/wp-content/uploads/Control-800.png
[link] => https://www.nextafter.com/experiments/how-using-an-iphone-notes-app-style-of-facebook-ad-creative-impact-clicks/attachment/control-png-8678/
[alt] =>
[author] => 0
[description] =>
[caption] =>
[name] => control-png-8678
[status] => inherit
[uploaded_to] => 64717
[date] => 2024-08-14 04:08:11
[modified] => 2024-08-14 04:08:11
[menu_order] => 0
[mime_type] => image/png
[type] => image
[subtype] => png
[icon] => https://www.nextafter.com/wp-includes/images/media/default.png
[width] => 2080
[height] => 1445
[sizes] => Array
(
[thumbnail] => https://www.nextafter.com/wp-content/uploads/Control-800-150x150.png
[thumbnail-width] => 150
[thumbnail-height] => 150
[medium] => https://www.nextafter.com/wp-content/uploads/Control-800-300x208.png
[medium-width] => 300
[medium-height] => 208
[medium_large] => https://www.nextafter.com/wp-content/uploads/Control-800-768x534.png
[medium_large-width] => 640
[medium_large-height] => 445
[large] => https://www.nextafter.com/wp-content/uploads/Control-800-1024x711.png
[large-width] => 640
[large-height] => 444
[1536x1536] => https://www.nextafter.com/wp-content/uploads/Control-800-1536x1067.png
[1536x1536-width] => 1536
[1536x1536-height] => 1067
[2048x2048] => https://www.nextafter.com/wp-content/uploads/Control-800-2048x1423.png
[2048x2048-width] => 2048
[2048x2048-height] => 1423
[tp-image-grid] => https://www.nextafter.com/wp-content/uploads/Control-800-700x700.png
[tp-image-grid-width] => 700
[tp-image-grid-height] => 700
[post-thumbnail] => https://www.nextafter.com/wp-content/uploads/Control-800-150x150.png
[post-thumbnail-width] => 150
[post-thumbnail-height] => 150
[admin] => https://www.nextafter.com/wp-content/uploads/Control-800-50x35.png
[admin-width] => 50
[admin-height] => 35
[full] => https://www.nextafter.com/wp-content/uploads/Control-800.png
[full-width] => 2080
[full-height] => 1445
[thumb_372] => https://www.nextafter.com/wp-content/uploads/Control-800-372x258.png
[thumb_372-width] => 372
[thumb_372-height] => 258
[single_post_thumbnail] => https://www.nextafter.com/wp-content/uploads/Control-800-900x625.png
[single_post_thumbnail-width] => 900
[single_post_thumbnail-height] => 625
[thumb_32_32] => https://www.nextafter.com/wp-content/uploads/Control-800-32x22.png
[thumb_32_32-width] => 32
[thumb_32_32-height] => 22
[thumb_48_48] => https://www.nextafter.com/wp-content/uploads/Control-800-48x33.png
[thumb_48_48-width] => 48
[thumb_48_48-height] => 33
[thumb_64_64] => https://www.nextafter.com/wp-content/uploads/Control-800-64x44.png
[thumb_64_64-width] => 64
[thumb_64_64-height] => 44
[thumb_128_128] => https://www.nextafter.com/wp-content/uploads/Control-800-128x89.png
[thumb_128_128-width] => 128
[thumb_128_128-height] => 89
[thumb_122_67] => https://www.nextafter.com/wp-content/uploads/Control-800-96x67.png
[thumb_122_67-width] => 96
[thumb_122_67-height] => 67
[thumb_153_153] => https://www.nextafter.com/wp-content/uploads/Control-800-153x106.png
[thumb_153_153-width] => 153
[thumb_153_153-height] => 106
[thumb_130_163] => https://www.nextafter.com/wp-content/uploads/Control-800-130x90.png
[thumb_130_163-width] => 130
[thumb_130_163-height] => 90
[thumb_232_177] => https://www.nextafter.com/wp-content/uploads/Control-800-232x161.png
[thumb_232_177-width] => 232
[thumb_232_177-height] => 161
[thumb_272_325] => https://www.nextafter.com/wp-content/uploads/Control-800-544x378.png
[thumb_272_325-width] => 544
[thumb_272_325-height] => 378
[thumb_271_346] => https://www.nextafter.com/wp-content/uploads/Control-800-271x188.png
[thumb_271_346-width] => 271
[thumb_271_346-height] => 188
[thumb_271_271] => https://www.nextafter.com/wp-content/uploads/Control-800-271x188.png
[thumb_271_271-width] => 271
[thumb_271_271-height] => 188
[thumb_272_172] => https://www.nextafter.com/wp-content/uploads/Control-800-248x172.png
[thumb_272_172-width] => 248
[thumb_272_172-height] => 172
[thumb_372_461] => https://www.nextafter.com/wp-content/uploads/Control-800-372x258.png
[thumb_372_461-width] => 372
[thumb_372_461-height] => 258
[thumb_372_500] => https://www.nextafter.com/wp-content/uploads/Control-800-372x258.png
[thumb_372_500-width] => 372
[thumb_372_500-height] => 258
[thumb_385_270] => https://www.nextafter.com/wp-content/uploads/Control-800-385x267.png
[thumb_385_270-width] => 385
[thumb_385_270-height] => 267
[thumb_416_241] => https://www.nextafter.com/wp-content/uploads/Control-800-347x241.png
[thumb_416_241-width] => 347
[thumb_416_241-height] => 241
[thumb_448_410] => https://www.nextafter.com/wp-content/uploads/Control-800-448x311.png
[thumb_448_410-width] => 448
[thumb_448_410-height] => 311
[thumb_448_410_retina] => https://www.nextafter.com/wp-content/uploads/Control-800-896x622.png
[thumb_448_410_retina-width] => 896
[thumb_448_410_retina-height] => 622
[thumb_574_382] => https://www.nextafter.com/wp-content/uploads/Control-800-550x382.png
[thumb_574_382-width] => 550
[thumb_574_382-height] => 382
[thumb_776_395] => https://www.nextafter.com/wp-content/uploads/Control-800-569x395.png
[thumb_776_395-width] => 569
[thumb_776_395-height] => 395
[gform-image-choice-sm] => https://www.nextafter.com/wp-content/uploads/Control-800.png
[gform-image-choice-sm-width] => 300
[gform-image-choice-sm-height] => 208
[gform-image-choice-md] => https://www.nextafter.com/wp-content/uploads/Control-800.png
[gform-image-choice-md-width] => 400
[gform-image-choice-md-height] => 278
[gform-image-choice-lg] => https://www.nextafter.com/wp-content/uploads/Control-800.png
[gform-image-choice-lg-width] => 600
[gform-image-choice-lg-height] => 417
)
)
[1] => Array
(
[ID] => 64740
[id] => 64740
[title] => Treatment_#1.png
[filename] => Treatment_1-202.png
[filesize] => 116457
[url] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202.png
[link] => https://www.nextafter.com/experiments/how-using-an-iphone-notes-app-style-of-facebook-ad-creative-impact-clicks/attachment/treatment_1-png-2963/
[alt] =>
[author] => 0
[description] =>
[caption] =>
[name] => treatment_1-png-2963
[status] => inherit
[uploaded_to] => 64717
[date] => 2024-08-14 04:08:25
[modified] => 2024-08-14 04:08:25
[menu_order] => 0
[mime_type] => image/png
[type] => image
[subtype] => png
[icon] => https://www.nextafter.com/wp-includes/images/media/default.png
[width] => 1080
[height] => 1080
[sizes] => Array
(
[thumbnail] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-150x150.png
[thumbnail-width] => 150
[thumbnail-height] => 150
[medium] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-300x300.png
[medium-width] => 300
[medium-height] => 300
[medium_large] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-768x768.png
[medium_large-width] => 640
[medium_large-height] => 640
[large] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-1024x1024.png
[large-width] => 640
[large-height] => 640
[1536x1536] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202.png
[1536x1536-width] => 1080
[1536x1536-height] => 1080
[2048x2048] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202.png
[2048x2048-width] => 1080
[2048x2048-height] => 1080
[tp-image-grid] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-700x700.png
[tp-image-grid-width] => 700
[tp-image-grid-height] => 700
[post-thumbnail] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-150x150.png
[post-thumbnail-width] => 150
[post-thumbnail-height] => 150
[admin] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-50x50.png
[admin-width] => 50
[admin-height] => 50
[full] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202.png
[full-width] => 1080
[full-height] => 1080
[thumb_372] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-372x372.png
[thumb_372-width] => 372
[thumb_372-height] => 372
[single_post_thumbnail] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-625x625.png
[single_post_thumbnail-width] => 625
[single_post_thumbnail-height] => 625
[thumb_32_32] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-32x32.png
[thumb_32_32-width] => 32
[thumb_32_32-height] => 32
[thumb_48_48] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-48x48.png
[thumb_48_48-width] => 48
[thumb_48_48-height] => 48
[thumb_64_64] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-64x64.png
[thumb_64_64-width] => 64
[thumb_64_64-height] => 64
[thumb_128_128] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-128x128.png
[thumb_128_128-width] => 128
[thumb_128_128-height] => 128
[thumb_122_67] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-67x67.png
[thumb_122_67-width] => 67
[thumb_122_67-height] => 67
[thumb_153_153] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-153x153.png
[thumb_153_153-width] => 153
[thumb_153_153-height] => 153
[thumb_130_163] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-130x130.png
[thumb_130_163-width] => 130
[thumb_130_163-height] => 130
[thumb_232_177] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-177x177.png
[thumb_232_177-width] => 177
[thumb_232_177-height] => 177
[thumb_272_325] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-544x544.png
[thumb_272_325-width] => 544
[thumb_272_325-height] => 544
[thumb_271_346] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-271x271.png
[thumb_271_346-width] => 271
[thumb_271_346-height] => 271
[thumb_271_271] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-271x271.png
[thumb_271_271-width] => 271
[thumb_271_271-height] => 271
[thumb_272_172] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-172x172.png
[thumb_272_172-width] => 172
[thumb_272_172-height] => 172
[thumb_372_461] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-372x372.png
[thumb_372_461-width] => 372
[thumb_372_461-height] => 372
[thumb_372_500] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-372x372.png
[thumb_372_500-width] => 372
[thumb_372_500-height] => 372
[thumb_385_270] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-270x270.png
[thumb_385_270-width] => 270
[thumb_385_270-height] => 270
[thumb_416_241] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-241x241.png
[thumb_416_241-width] => 241
[thumb_416_241-height] => 241
[thumb_448_410] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-410x410.png
[thumb_448_410-width] => 410
[thumb_448_410-height] => 410
[thumb_448_410_retina] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-820x820.png
[thumb_448_410_retina-width] => 820
[thumb_448_410_retina-height] => 820
[thumb_574_382] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-382x382.png
[thumb_574_382-width] => 382
[thumb_574_382-height] => 382
[thumb_776_395] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-395x395.png
[thumb_776_395-width] => 395
[thumb_776_395-height] => 395
[gform-image-choice-sm] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202.png
[gform-image-choice-sm-width] => 300
[gform-image-choice-sm-height] => 300
[gform-image-choice-md] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202.png
[gform-image-choice-md-width] => 400
[gform-image-choice-md-height] => 400
[gform-image-choice-lg] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202.png
[gform-image-choice-lg-width] => 600
[gform-image-choice-lg-height] => 600
)
)
)
T1: Treatment #1
Array
(
[0] => Array
(
[ID] => 64736
[id] => 64736
[title] => Control.png
[filename] => Control-800.png
[filesize] => 187041
[url] => https://www.nextafter.com/wp-content/uploads/Control-800.png
[link] => https://www.nextafter.com/experiments/how-using-an-iphone-notes-app-style-of-facebook-ad-creative-impact-clicks/attachment/control-png-8678/
[alt] =>
[author] => 0
[description] =>
[caption] =>
[name] => control-png-8678
[status] => inherit
[uploaded_to] => 64717
[date] => 2024-08-14 04:08:11
[modified] => 2024-08-14 04:08:11
[menu_order] => 0
[mime_type] => image/png
[type] => image
[subtype] => png
[icon] => https://www.nextafter.com/wp-includes/images/media/default.png
[width] => 2080
[height] => 1445
[sizes] => Array
(
[thumbnail] => https://www.nextafter.com/wp-content/uploads/Control-800-150x150.png
[thumbnail-width] => 150
[thumbnail-height] => 150
[medium] => https://www.nextafter.com/wp-content/uploads/Control-800-300x208.png
[medium-width] => 300
[medium-height] => 208
[medium_large] => https://www.nextafter.com/wp-content/uploads/Control-800-768x534.png
[medium_large-width] => 640
[medium_large-height] => 445
[large] => https://www.nextafter.com/wp-content/uploads/Control-800-1024x711.png
[large-width] => 640
[large-height] => 444
[1536x1536] => https://www.nextafter.com/wp-content/uploads/Control-800-1536x1067.png
[1536x1536-width] => 1536
[1536x1536-height] => 1067
[2048x2048] => https://www.nextafter.com/wp-content/uploads/Control-800-2048x1423.png
[2048x2048-width] => 2048
[2048x2048-height] => 1423
[tp-image-grid] => https://www.nextafter.com/wp-content/uploads/Control-800-700x700.png
[tp-image-grid-width] => 700
[tp-image-grid-height] => 700
[post-thumbnail] => https://www.nextafter.com/wp-content/uploads/Control-800-150x150.png
[post-thumbnail-width] => 150
[post-thumbnail-height] => 150
[admin] => https://www.nextafter.com/wp-content/uploads/Control-800-50x35.png
[admin-width] => 50
[admin-height] => 35
[full] => https://www.nextafter.com/wp-content/uploads/Control-800.png
[full-width] => 2080
[full-height] => 1445
[thumb_372] => https://www.nextafter.com/wp-content/uploads/Control-800-372x258.png
[thumb_372-width] => 372
[thumb_372-height] => 258
[single_post_thumbnail] => https://www.nextafter.com/wp-content/uploads/Control-800-900x625.png
[single_post_thumbnail-width] => 900
[single_post_thumbnail-height] => 625
[thumb_32_32] => https://www.nextafter.com/wp-content/uploads/Control-800-32x22.png
[thumb_32_32-width] => 32
[thumb_32_32-height] => 22
[thumb_48_48] => https://www.nextafter.com/wp-content/uploads/Control-800-48x33.png
[thumb_48_48-width] => 48
[thumb_48_48-height] => 33
[thumb_64_64] => https://www.nextafter.com/wp-content/uploads/Control-800-64x44.png
[thumb_64_64-width] => 64
[thumb_64_64-height] => 44
[thumb_128_128] => https://www.nextafter.com/wp-content/uploads/Control-800-128x89.png
[thumb_128_128-width] => 128
[thumb_128_128-height] => 89
[thumb_122_67] => https://www.nextafter.com/wp-content/uploads/Control-800-96x67.png
[thumb_122_67-width] => 96
[thumb_122_67-height] => 67
[thumb_153_153] => https://www.nextafter.com/wp-content/uploads/Control-800-153x106.png
[thumb_153_153-width] => 153
[thumb_153_153-height] => 106
[thumb_130_163] => https://www.nextafter.com/wp-content/uploads/Control-800-130x90.png
[thumb_130_163-width] => 130
[thumb_130_163-height] => 90
[thumb_232_177] => https://www.nextafter.com/wp-content/uploads/Control-800-232x161.png
[thumb_232_177-width] => 232
[thumb_232_177-height] => 161
[thumb_272_325] => https://www.nextafter.com/wp-content/uploads/Control-800-544x378.png
[thumb_272_325-width] => 544
[thumb_272_325-height] => 378
[thumb_271_346] => https://www.nextafter.com/wp-content/uploads/Control-800-271x188.png
[thumb_271_346-width] => 271
[thumb_271_346-height] => 188
[thumb_271_271] => https://www.nextafter.com/wp-content/uploads/Control-800-271x188.png
[thumb_271_271-width] => 271
[thumb_271_271-height] => 188
[thumb_272_172] => https://www.nextafter.com/wp-content/uploads/Control-800-248x172.png
[thumb_272_172-width] => 248
[thumb_272_172-height] => 172
[thumb_372_461] => https://www.nextafter.com/wp-content/uploads/Control-800-372x258.png
[thumb_372_461-width] => 372
[thumb_372_461-height] => 258
[thumb_372_500] => https://www.nextafter.com/wp-content/uploads/Control-800-372x258.png
[thumb_372_500-width] => 372
[thumb_372_500-height] => 258
[thumb_385_270] => https://www.nextafter.com/wp-content/uploads/Control-800-385x267.png
[thumb_385_270-width] => 385
[thumb_385_270-height] => 267
[thumb_416_241] => https://www.nextafter.com/wp-content/uploads/Control-800-347x241.png
[thumb_416_241-width] => 347
[thumb_416_241-height] => 241
[thumb_448_410] => https://www.nextafter.com/wp-content/uploads/Control-800-448x311.png
[thumb_448_410-width] => 448
[thumb_448_410-height] => 311
[thumb_448_410_retina] => https://www.nextafter.com/wp-content/uploads/Control-800-896x622.png
[thumb_448_410_retina-width] => 896
[thumb_448_410_retina-height] => 622
[thumb_574_382] => https://www.nextafter.com/wp-content/uploads/Control-800-550x382.png
[thumb_574_382-width] => 550
[thumb_574_382-height] => 382
[thumb_776_395] => https://www.nextafter.com/wp-content/uploads/Control-800-569x395.png
[thumb_776_395-width] => 569
[thumb_776_395-height] => 395
[gform-image-choice-sm] => https://www.nextafter.com/wp-content/uploads/Control-800.png
[gform-image-choice-sm-width] => 300
[gform-image-choice-sm-height] => 208
[gform-image-choice-md] => https://www.nextafter.com/wp-content/uploads/Control-800.png
[gform-image-choice-md-width] => 400
[gform-image-choice-md-height] => 278
[gform-image-choice-lg] => https://www.nextafter.com/wp-content/uploads/Control-800.png
[gform-image-choice-lg-width] => 600
[gform-image-choice-lg-height] => 417
)
)
[1] => Array
(
[ID] => 64740
[id] => 64740
[title] => Treatment_#1.png
[filename] => Treatment_1-202.png
[filesize] => 116457
[url] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202.png
[link] => https://www.nextafter.com/experiments/how-using-an-iphone-notes-app-style-of-facebook-ad-creative-impact-clicks/attachment/treatment_1-png-2963/
[alt] =>
[author] => 0
[description] =>
[caption] =>
[name] => treatment_1-png-2963
[status] => inherit
[uploaded_to] => 64717
[date] => 2024-08-14 04:08:25
[modified] => 2024-08-14 04:08:25
[menu_order] => 0
[mime_type] => image/png
[type] => image
[subtype] => png
[icon] => https://www.nextafter.com/wp-includes/images/media/default.png
[width] => 1080
[height] => 1080
[sizes] => Array
(
[thumbnail] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-150x150.png
[thumbnail-width] => 150
[thumbnail-height] => 150
[medium] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-300x300.png
[medium-width] => 300
[medium-height] => 300
[medium_large] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-768x768.png
[medium_large-width] => 640
[medium_large-height] => 640
[large] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-1024x1024.png
[large-width] => 640
[large-height] => 640
[1536x1536] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202.png
[1536x1536-width] => 1080
[1536x1536-height] => 1080
[2048x2048] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202.png
[2048x2048-width] => 1080
[2048x2048-height] => 1080
[tp-image-grid] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-700x700.png
[tp-image-grid-width] => 700
[tp-image-grid-height] => 700
[post-thumbnail] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-150x150.png
[post-thumbnail-width] => 150
[post-thumbnail-height] => 150
[admin] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-50x50.png
[admin-width] => 50
[admin-height] => 50
[full] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202.png
[full-width] => 1080
[full-height] => 1080
[thumb_372] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-372x372.png
[thumb_372-width] => 372
[thumb_372-height] => 372
[single_post_thumbnail] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-625x625.png
[single_post_thumbnail-width] => 625
[single_post_thumbnail-height] => 625
[thumb_32_32] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-32x32.png
[thumb_32_32-width] => 32
[thumb_32_32-height] => 32
[thumb_48_48] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-48x48.png
[thumb_48_48-width] => 48
[thumb_48_48-height] => 48
[thumb_64_64] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-64x64.png
[thumb_64_64-width] => 64
[thumb_64_64-height] => 64
[thumb_128_128] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-128x128.png
[thumb_128_128-width] => 128
[thumb_128_128-height] => 128
[thumb_122_67] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-67x67.png
[thumb_122_67-width] => 67
[thumb_122_67-height] => 67
[thumb_153_153] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-153x153.png
[thumb_153_153-width] => 153
[thumb_153_153-height] => 153
[thumb_130_163] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-130x130.png
[thumb_130_163-width] => 130
[thumb_130_163-height] => 130
[thumb_232_177] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-177x177.png
[thumb_232_177-width] => 177
[thumb_232_177-height] => 177
[thumb_272_325] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-544x544.png
[thumb_272_325-width] => 544
[thumb_272_325-height] => 544
[thumb_271_346] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-271x271.png
[thumb_271_346-width] => 271
[thumb_271_346-height] => 271
[thumb_271_271] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-271x271.png
[thumb_271_271-width] => 271
[thumb_271_271-height] => 271
[thumb_272_172] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-172x172.png
[thumb_272_172-width] => 172
[thumb_272_172-height] => 172
[thumb_372_461] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-372x372.png
[thumb_372_461-width] => 372
[thumb_372_461-height] => 372
[thumb_372_500] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-372x372.png
[thumb_372_500-width] => 372
[thumb_372_500-height] => 372
[thumb_385_270] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-270x270.png
[thumb_385_270-width] => 270
[thumb_385_270-height] => 270
[thumb_416_241] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-241x241.png
[thumb_416_241-width] => 241
[thumb_416_241-height] => 241
[thumb_448_410] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-410x410.png
[thumb_448_410-width] => 410
[thumb_448_410-height] => 410
[thumb_448_410_retina] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-820x820.png
[thumb_448_410_retina-width] => 820
[thumb_448_410_retina-height] => 820
[thumb_574_382] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-382x382.png
[thumb_574_382-width] => 382
[thumb_574_382-height] => 382
[thumb_776_395] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202-395x395.png
[thumb_776_395-width] => 395
[thumb_776_395-height] => 395
[gform-image-choice-sm] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202.png
[gform-image-choice-sm-width] => 300
[gform-image-choice-sm-height] => 300
[gform-image-choice-md] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202.png
[gform-image-choice-md-width] => 400
[gform-image-choice-md-height] => 400
[gform-image-choice-lg] => https://www.nextafter.com/wp-content/uploads/Treatment_1-202.png
[gform-image-choice-lg-width] => 600
[gform-image-choice-lg-height] => 600
)
)
)
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.50% | ||
| T1: | Treatment #1 | 0.41% | -17.9% | 96.5% |
This experiment has a required sample size of 43,178 in order to be valid. Since the experiment had a total sample size of 101,600, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
17.9% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
In this experiment, we anticipated that the iPhone notes app style ad would outperform the organic-style text ad, leveraging its familiar and visually engaging format. However, the results showed the opposite, with the Control ads (organic-style text) achieving better performance.
The Treatment ads led to a -17.9% decrease in clicks compared to the Control ads, with a 96.5% level of confidence (LOC). The Control ads had a higher click rate (0.50%) compared to the Treatment ads (0.41%).
This test did reach statistical validity, indicating that the iPhone notes app style ad was less effective at driving clicks. Despite our hypothesis, the Treatment ads may not have resonated as strongly with the audience.
We theorized it could be due to visual engagement. The Treatment ads, designed to mimic the iPhone notes app, may have lacked the necessary visual contrast to stand out in users’ Facebook feeds. The minimalistic design, while intended to appear authentic, could have been perceived as bland or too subtle, leading users to overlook the ad. The Control, while also very minimal in design was mostly in a deep blue color that can stand out more clearly.