let us help
Subscribe
NextAfter

How using an iPhone Notes App style of Facebook ad creative impacted clicks to a video series landing page

Experiment ID: #183560

NextAfter

Experiment Summary

Timeframe: 06/07/2024 - 06/24/2024

Focus on the Family is a large Christian ministry organization that creates and shares free resources like video series to encourage families. They promoted these free video series on Facebook and wanted to see if we could try some radically different ad creatives to improve clicks and email signups from their ads.

For this experiment, we tested using an organic, non-branded ad that resembles a screenshot of someone’s iPhone Notes App. This ad creative featured an impact-focused headline and three bullet point statements below.

The control ads for this test used the same headline and bullet points, but they would look like the typical branded ads we usually run on Facebook.

Our hypothesis is that the Notes App ads, with their authentic and eye-catching design, will significantly boost clicks and email signups. This is due to their unique appearance, which we believe will resonate more with our target audience on Facebook.

We ran this as an A/B campaigns test in Facebook with the ad budget split evenly. The Control campaign had the standard ads for two video series, and the Treatment campaign had the Notes App version of the ad creative for those same two video series. All other variables remained the same, and the test ran for a total of 2.5 weeks.

Research Question

We believe that changing the ad creative image to mimic a Notes App style for Facebook traffic will achieve an increase in clicks and email signups because the ad creative looks more authentic and eye-catching. because the ad creative looks more authentic and eye-catching. because the ad creative looks more authentic and eye-catching. because the ad creative looks more authentic and eye-catching.

Design

C: Control - Standard Design
Array ( [0] => Array ( [ID] => 64723 [id] => 64723 [title] => Control_-_Standard_Design.png [filename] => Control_-_Standard_Design-1.png [filesize] => 200785 [url] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [link] => https://www.nextafter.com/experiments/how-using-an-iphone-notes-app-style-of-facebook-ad-creative-impacted-clicks-to-a-video-series-landing-page/attachment/control_-_standard_design-png-2/ [alt] => [author] => 0 [description] => [caption] => [name] => control_-_standard_design-png-2 [status] => inherit [uploaded_to] => 60289 [date] => 2024-08-14 04:06:51 [modified] => 2024-08-14 04:06:51 [menu_order] => 0 [mime_type] => image/png [type] => image [subtype] => png [icon] => https://www.nextafter.com/wp-includes/images/media/default.png [width] => 356 [height] => 586 [sizes] => Array ( [thumbnail] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-150x150.png [thumbnail-width] => 150 [thumbnail-height] => 150 [medium] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-182x300.png [medium-width] => 182 [medium-height] => 300 [medium_large] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [medium_large-width] => 356 [medium_large-height] => 586 [large] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [large-width] => 356 [large-height] => 586 [1536x1536] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [1536x1536-width] => 356 [1536x1536-height] => 586 [2048x2048] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [2048x2048-width] => 356 [2048x2048-height] => 586 [tp-image-grid] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-700x700.png [tp-image-grid-width] => 700 [tp-image-grid-height] => 700 [post-thumbnail] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-150x150.png [post-thumbnail-width] => 150 [post-thumbnail-height] => 150 [admin] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-30x50.png [admin-width] => 30 [admin-height] => 50 [full] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [full-width] => 356 [full-height] => 586 [thumb_372] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [thumb_372-width] => 356 [thumb_372-height] => 586 [single_post_thumbnail] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [single_post_thumbnail-width] => 356 [single_post_thumbnail-height] => 586 [thumb_32_32] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-19x32.png [thumb_32_32-width] => 19 [thumb_32_32-height] => 32 [thumb_48_48] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-29x48.png [thumb_48_48-width] => 29 [thumb_48_48-height] => 48 [thumb_64_64] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-39x64.png [thumb_64_64-width] => 39 [thumb_64_64-height] => 64 [thumb_128_128] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-78x128.png [thumb_128_128-width] => 78 [thumb_128_128-height] => 128 [thumb_122_67] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-41x67.png [thumb_122_67-width] => 41 [thumb_122_67-height] => 67 [thumb_153_153] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-93x153.png [thumb_153_153-width] => 93 [thumb_153_153-height] => 153 [thumb_130_163] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-99x163.png [thumb_130_163-width] => 99 [thumb_130_163-height] => 163 [thumb_232_177] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-108x177.png [thumb_232_177-width] => 108 [thumb_232_177-height] => 177 [thumb_272_325] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [thumb_272_325-width] => 356 [thumb_272_325-height] => 586 [thumb_271_346] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-210x346.png [thumb_271_346-width] => 210 [thumb_271_346-height] => 346 [thumb_271_271] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-165x271.png [thumb_271_271-width] => 165 [thumb_271_271-height] => 271 [thumb_272_172] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-104x172.png [thumb_272_172-width] => 104 [thumb_272_172-height] => 172 [thumb_372_461] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-280x461.png [thumb_372_461-width] => 280 [thumb_372_461-height] => 461 [thumb_372_500] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-304x500.png [thumb_372_500-width] => 304 [thumb_372_500-height] => 500 [thumb_385_270] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-164x270.png [thumb_385_270-width] => 164 [thumb_385_270-height] => 270 [thumb_416_241] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-146x241.png [thumb_416_241-width] => 146 [thumb_416_241-height] => 241 [thumb_448_410] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-249x410.png [thumb_448_410-width] => 249 [thumb_448_410-height] => 410 [thumb_448_410_retina] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [thumb_448_410_retina-width] => 356 [thumb_448_410_retina-height] => 586 [thumb_574_382] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-232x382.png [thumb_574_382-width] => 232 [thumb_574_382-height] => 382 [thumb_776_395] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-240x395.png [thumb_776_395-width] => 240 [thumb_776_395-height] => 395 [gform-image-choice-sm] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [gform-image-choice-sm-width] => 182 [gform-image-choice-sm-height] => 300 [gform-image-choice-md] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [gform-image-choice-md-width] => 243 [gform-image-choice-md-height] => 400 [gform-image-choice-lg] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [gform-image-choice-lg-width] => 356 [gform-image-choice-lg-height] => 586 ) ) [1] => Array ( [ID] => 64724 [id] => 64724 [title] => Treatment_-_Notes_App_Design.png [filename] => Treatment_-_Notes_App_Design-1.png [filesize] => 90693 [url] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [link] => https://www.nextafter.com/experiments/how-using-an-iphone-notes-app-style-of-facebook-ad-creative-impacted-clicks-to-a-video-series-landing-page/attachment/treatment_-_notes_app_design-png-2/ [alt] => [author] => 0 [description] => [caption] => [name] => treatment_-_notes_app_design-png-2 [status] => inherit [uploaded_to] => 60289 [date] => 2024-08-14 04:06:56 [modified] => 2024-08-14 04:06:56 [menu_order] => 0 [mime_type] => image/png [type] => image [subtype] => png [icon] => https://www.nextafter.com/wp-includes/images/media/default.png [width] => 356 [height] => 586 [sizes] => Array ( [thumbnail] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-150x150.png [thumbnail-width] => 150 [thumbnail-height] => 150 [medium] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-182x300.png [medium-width] => 182 [medium-height] => 300 [medium_large] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [medium_large-width] => 356 [medium_large-height] => 586 [large] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [large-width] => 356 [large-height] => 586 [1536x1536] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [1536x1536-width] => 356 [1536x1536-height] => 586 [2048x2048] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [2048x2048-width] => 356 [2048x2048-height] => 586 [tp-image-grid] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-700x700.png [tp-image-grid-width] => 700 [tp-image-grid-height] => 700 [post-thumbnail] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-150x150.png [post-thumbnail-width] => 150 [post-thumbnail-height] => 150 [admin] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-30x50.png [admin-width] => 30 [admin-height] => 50 [full] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [full-width] => 356 [full-height] => 586 [thumb_372] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [thumb_372-width] => 356 [thumb_372-height] => 586 [single_post_thumbnail] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [single_post_thumbnail-width] => 356 [single_post_thumbnail-height] => 586 [thumb_32_32] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-19x32.png [thumb_32_32-width] => 19 [thumb_32_32-height] => 32 [thumb_48_48] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-29x48.png [thumb_48_48-width] => 29 [thumb_48_48-height] => 48 [thumb_64_64] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-39x64.png [thumb_64_64-width] => 39 [thumb_64_64-height] => 64 [thumb_128_128] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-78x128.png [thumb_128_128-width] => 78 [thumb_128_128-height] => 128 [thumb_122_67] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-41x67.png [thumb_122_67-width] => 41 [thumb_122_67-height] => 67 [thumb_153_153] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-93x153.png [thumb_153_153-width] => 93 [thumb_153_153-height] => 153 [thumb_130_163] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-99x163.png [thumb_130_163-width] => 99 [thumb_130_163-height] => 163 [thumb_232_177] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-108x177.png [thumb_232_177-width] => 108 [thumb_232_177-height] => 177 [thumb_272_325] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [thumb_272_325-width] => 356 [thumb_272_325-height] => 586 [thumb_271_346] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-210x346.png [thumb_271_346-width] => 210 [thumb_271_346-height] => 346 [thumb_271_271] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-165x271.png [thumb_271_271-width] => 165 [thumb_271_271-height] => 271 [thumb_272_172] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-104x172.png [thumb_272_172-width] => 104 [thumb_272_172-height] => 172 [thumb_372_461] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-280x461.png [thumb_372_461-width] => 280 [thumb_372_461-height] => 461 [thumb_372_500] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-304x500.png [thumb_372_500-width] => 304 [thumb_372_500-height] => 500 [thumb_385_270] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-164x270.png [thumb_385_270-width] => 164 [thumb_385_270-height] => 270 [thumb_416_241] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-146x241.png [thumb_416_241-width] => 146 [thumb_416_241-height] => 241 [thumb_448_410] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-249x410.png [thumb_448_410-width] => 249 [thumb_448_410-height] => 410 [thumb_448_410_retina] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [thumb_448_410_retina-width] => 356 [thumb_448_410_retina-height] => 586 [thumb_574_382] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-232x382.png [thumb_574_382-width] => 232 [thumb_574_382-height] => 382 [thumb_776_395] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-240x395.png [thumb_776_395-width] => 240 [thumb_776_395-height] => 395 [gform-image-choice-sm] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [gform-image-choice-sm-width] => 182 [gform-image-choice-sm-height] => 300 [gform-image-choice-md] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [gform-image-choice-md-width] => 243 [gform-image-choice-md-height] => 400 [gform-image-choice-lg] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [gform-image-choice-lg-width] => 356 [gform-image-choice-lg-height] => 586 ) ) )
T1: Treatment - Notes App Design
Array ( [0] => Array ( [ID] => 64723 [id] => 64723 [title] => Control_-_Standard_Design.png [filename] => Control_-_Standard_Design-1.png [filesize] => 200785 [url] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [link] => https://www.nextafter.com/experiments/how-using-an-iphone-notes-app-style-of-facebook-ad-creative-impacted-clicks-to-a-video-series-landing-page/attachment/control_-_standard_design-png-2/ [alt] => [author] => 0 [description] => [caption] => [name] => control_-_standard_design-png-2 [status] => inherit [uploaded_to] => 60289 [date] => 2024-08-14 04:06:51 [modified] => 2024-08-14 04:06:51 [menu_order] => 0 [mime_type] => image/png [type] => image [subtype] => png [icon] => https://www.nextafter.com/wp-includes/images/media/default.png [width] => 356 [height] => 586 [sizes] => Array ( [thumbnail] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-150x150.png [thumbnail-width] => 150 [thumbnail-height] => 150 [medium] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-182x300.png [medium-width] => 182 [medium-height] => 300 [medium_large] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [medium_large-width] => 356 [medium_large-height] => 586 [large] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [large-width] => 356 [large-height] => 586 [1536x1536] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [1536x1536-width] => 356 [1536x1536-height] => 586 [2048x2048] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [2048x2048-width] => 356 [2048x2048-height] => 586 [tp-image-grid] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-700x700.png [tp-image-grid-width] => 700 [tp-image-grid-height] => 700 [post-thumbnail] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-150x150.png [post-thumbnail-width] => 150 [post-thumbnail-height] => 150 [admin] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-30x50.png [admin-width] => 30 [admin-height] => 50 [full] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [full-width] => 356 [full-height] => 586 [thumb_372] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [thumb_372-width] => 356 [thumb_372-height] => 586 [single_post_thumbnail] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [single_post_thumbnail-width] => 356 [single_post_thumbnail-height] => 586 [thumb_32_32] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-19x32.png [thumb_32_32-width] => 19 [thumb_32_32-height] => 32 [thumb_48_48] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-29x48.png [thumb_48_48-width] => 29 [thumb_48_48-height] => 48 [thumb_64_64] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-39x64.png [thumb_64_64-width] => 39 [thumb_64_64-height] => 64 [thumb_128_128] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-78x128.png [thumb_128_128-width] => 78 [thumb_128_128-height] => 128 [thumb_122_67] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-41x67.png [thumb_122_67-width] => 41 [thumb_122_67-height] => 67 [thumb_153_153] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-93x153.png [thumb_153_153-width] => 93 [thumb_153_153-height] => 153 [thumb_130_163] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-99x163.png [thumb_130_163-width] => 99 [thumb_130_163-height] => 163 [thumb_232_177] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-108x177.png [thumb_232_177-width] => 108 [thumb_232_177-height] => 177 [thumb_272_325] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [thumb_272_325-width] => 356 [thumb_272_325-height] => 586 [thumb_271_346] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-210x346.png [thumb_271_346-width] => 210 [thumb_271_346-height] => 346 [thumb_271_271] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-165x271.png [thumb_271_271-width] => 165 [thumb_271_271-height] => 271 [thumb_272_172] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-104x172.png [thumb_272_172-width] => 104 [thumb_272_172-height] => 172 [thumb_372_461] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-280x461.png [thumb_372_461-width] => 280 [thumb_372_461-height] => 461 [thumb_372_500] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-304x500.png [thumb_372_500-width] => 304 [thumb_372_500-height] => 500 [thumb_385_270] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-164x270.png [thumb_385_270-width] => 164 [thumb_385_270-height] => 270 [thumb_416_241] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-146x241.png [thumb_416_241-width] => 146 [thumb_416_241-height] => 241 [thumb_448_410] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-249x410.png [thumb_448_410-width] => 249 [thumb_448_410-height] => 410 [thumb_448_410_retina] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [thumb_448_410_retina-width] => 356 [thumb_448_410_retina-height] => 586 [thumb_574_382] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-232x382.png [thumb_574_382-width] => 232 [thumb_574_382-height] => 382 [thumb_776_395] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1-240x395.png [thumb_776_395-width] => 240 [thumb_776_395-height] => 395 [gform-image-choice-sm] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [gform-image-choice-sm-width] => 182 [gform-image-choice-sm-height] => 300 [gform-image-choice-md] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [gform-image-choice-md-width] => 243 [gform-image-choice-md-height] => 400 [gform-image-choice-lg] => https://www.nextafter.com/wp-content/uploads/Control_-_Standard_Design-1.png [gform-image-choice-lg-width] => 356 [gform-image-choice-lg-height] => 586 ) ) [1] => Array ( [ID] => 64724 [id] => 64724 [title] => Treatment_-_Notes_App_Design.png [filename] => Treatment_-_Notes_App_Design-1.png [filesize] => 90693 [url] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [link] => https://www.nextafter.com/experiments/how-using-an-iphone-notes-app-style-of-facebook-ad-creative-impacted-clicks-to-a-video-series-landing-page/attachment/treatment_-_notes_app_design-png-2/ [alt] => [author] => 0 [description] => [caption] => [name] => treatment_-_notes_app_design-png-2 [status] => inherit [uploaded_to] => 60289 [date] => 2024-08-14 04:06:56 [modified] => 2024-08-14 04:06:56 [menu_order] => 0 [mime_type] => image/png [type] => image [subtype] => png [icon] => https://www.nextafter.com/wp-includes/images/media/default.png [width] => 356 [height] => 586 [sizes] => Array ( [thumbnail] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-150x150.png [thumbnail-width] => 150 [thumbnail-height] => 150 [medium] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-182x300.png [medium-width] => 182 [medium-height] => 300 [medium_large] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [medium_large-width] => 356 [medium_large-height] => 586 [large] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [large-width] => 356 [large-height] => 586 [1536x1536] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [1536x1536-width] => 356 [1536x1536-height] => 586 [2048x2048] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [2048x2048-width] => 356 [2048x2048-height] => 586 [tp-image-grid] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-700x700.png [tp-image-grid-width] => 700 [tp-image-grid-height] => 700 [post-thumbnail] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-150x150.png [post-thumbnail-width] => 150 [post-thumbnail-height] => 150 [admin] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-30x50.png [admin-width] => 30 [admin-height] => 50 [full] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [full-width] => 356 [full-height] => 586 [thumb_372] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [thumb_372-width] => 356 [thumb_372-height] => 586 [single_post_thumbnail] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [single_post_thumbnail-width] => 356 [single_post_thumbnail-height] => 586 [thumb_32_32] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-19x32.png [thumb_32_32-width] => 19 [thumb_32_32-height] => 32 [thumb_48_48] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-29x48.png [thumb_48_48-width] => 29 [thumb_48_48-height] => 48 [thumb_64_64] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-39x64.png [thumb_64_64-width] => 39 [thumb_64_64-height] => 64 [thumb_128_128] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-78x128.png [thumb_128_128-width] => 78 [thumb_128_128-height] => 128 [thumb_122_67] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-41x67.png [thumb_122_67-width] => 41 [thumb_122_67-height] => 67 [thumb_153_153] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-93x153.png [thumb_153_153-width] => 93 [thumb_153_153-height] => 153 [thumb_130_163] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-99x163.png [thumb_130_163-width] => 99 [thumb_130_163-height] => 163 [thumb_232_177] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-108x177.png [thumb_232_177-width] => 108 [thumb_232_177-height] => 177 [thumb_272_325] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [thumb_272_325-width] => 356 [thumb_272_325-height] => 586 [thumb_271_346] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-210x346.png [thumb_271_346-width] => 210 [thumb_271_346-height] => 346 [thumb_271_271] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-165x271.png [thumb_271_271-width] => 165 [thumb_271_271-height] => 271 [thumb_272_172] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-104x172.png [thumb_272_172-width] => 104 [thumb_272_172-height] => 172 [thumb_372_461] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-280x461.png [thumb_372_461-width] => 280 [thumb_372_461-height] => 461 [thumb_372_500] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-304x500.png [thumb_372_500-width] => 304 [thumb_372_500-height] => 500 [thumb_385_270] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-164x270.png [thumb_385_270-width] => 164 [thumb_385_270-height] => 270 [thumb_416_241] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-146x241.png [thumb_416_241-width] => 146 [thumb_416_241-height] => 241 [thumb_448_410] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-249x410.png [thumb_448_410-width] => 249 [thumb_448_410-height] => 410 [thumb_448_410_retina] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [thumb_448_410_retina-width] => 356 [thumb_448_410_retina-height] => 586 [thumb_574_382] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-232x382.png [thumb_574_382-width] => 232 [thumb_574_382-height] => 382 [thumb_776_395] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1-240x395.png [thumb_776_395-width] => 240 [thumb_776_395-height] => 395 [gform-image-choice-sm] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [gform-image-choice-sm-width] => 182 [gform-image-choice-sm-height] => 300 [gform-image-choice-md] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [gform-image-choice-md-width] => 243 [gform-image-choice-md-height] => 400 [gform-image-choice-lg] => https://www.nextafter.com/wp-content/uploads/Treatment_-_Notes_App_Design-1.png [gform-image-choice-lg-width] => 356 [gform-image-choice-lg-height] => 586 ) ) )

Results

  Treatment Name Click Rate Relative Difference Confidence
C: Control - Standard Design 0.79%
T1: Treatment - Notes App Design 0.74% -6.1% 100.0%

This experiment has a required sample size of 253,145 in order to be valid. Since the experiment had a total sample size of 2,069,300, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    6.1% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

We expected the Notes App style of ad creative to perform better in this test, and we were surprised to see that our Control ads actually had the best results.

These are the results we saw:

  • The Treatment ads led to a -6.1% decrease in clicks to the video series landing page, with a 99.9% level of confidence (LOC).
  • The Treatment ads also validated a -7.9% decrease in the number of emails acquired, with a 99.6% LOC.
  • And the Treatment ads had a -24.0% decrease in donations, with a 98.8% LOC.
  • The Control ads had a campaign ROI of 1.39, which was higher than the Treatment campaign's ROI of 0.94.

Our key learnings from this experiment are that the Notes App style ad creative did not perform as well as the standard branded ads in terms of clicks, email signups, and donations. Despite the hypothesis that the more authentic and eye-catching design would lead to an increase in engagement, the results showed a decrease in clicks to the landing page, emails acquired, and donations.

We have a couple of theories on why the Treatment ads performed worse in this test, and these are ideas that we can test into with future ads:

  1. The Treatment ads lacked the Focus on the Family branding and logo, which could have decreased their credibility
  2. The Treatment ads did not incorporate any imagery to support the value or impact of the video series, but the Control ads used imagery to support their claims.
  3. Ex: For the "Reconnected" marriage video series ad, the image in the Control is a couple holding hands. This supports the message that the video series will help the reader reconnect with their spouse.
  4. The Control ads had more color to help them stand out on Facebook, whereas the white background of the Treatment ads caused them to almost "blend in" with the rest of the white space on the page.
  5. Ex: For the "Reconnected" marriage video series ad, the image in the Control is a couple holding hands. This supports the message that the video series will help the reader reconnect with their spouse.

Experiment Documented by NextAfter

Question about experiment #183560

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.