How removing images on an email newsletter impacts click through rates.
WUNC
Experiment Summary
Timeframe: 01/25/2024 - 01/26/2024
WUNC is North Carolina’s largest NPR station, and they have a weekly enewsletter featuring some of the top stories of the week that deploys every Thursday.
For this test, we wanted to see whether removing images and making the email look more like a human sent it made an impact on click through rates.
Research Question
We believe that removing images for enewsletter readers will achieve a change in click through rates.
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | With Images | 8.3% | ||
| T1: | Without Images | 6.8% | -17.3% | 100.0% |
This experiment has a required sample size of 2,612 in order to be valid. Since the experiment had a total sample size of 29,791, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
17.3% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The key learning from this experiment is that removing images from the enewsletter resulted in a decrease in click-through rates. The treatment group, which received the enewsletter without images, had a 17.3% decrease in clicks compared to the control group.
Based on this result, it is evident that images play an important role in engaging readers and encouraging them to click on the content. Therefore, in future experimentation, it is recommended to continue including images in the enewsletter. However, it may be worth exploring different types of images or optimizing the placement and design of the images to maximize their impact on click-through rates.
Additionally, it is important to consider other factors that may have influenced the results. The experiment ran for only 1 day, which may not have allowed enough time for readers to adapt to the new format. Conducting experiments over a longer duration could provide more reliable insights.
Overall, the findings suggest that images are a valuable component of the enewsletter and should be used strategically to enhance reader engagement and increase click-through rates.