Illinois Policy Institute

How testing different Dear Reader survey topics impacted email acquisition

Experiment ID: #152110

Illinois Policy Institute

Experiment Summary

Timeframe: 05/17/2023 - 06/10/2023

IPI has two active surveys: State of Our State and School Choice. We want to see which survey topic is most appealing to our reader audience and can collect the most emails, so we’ve separated them via a 50/50 Dear Reader experiment.

These Dear Readers are implemented at the end of policy research articles, so 50% of readers will be asked to complete the State of Our State Survey and 50% will be asked to complete the School Choice Survey once they are done reading their respective article.

Let’s see which version acquires the most emails.

Research Question

We believe that implementing a dear reader asking for opinions on the active topic of school choice in Illinois for page visitors will achieve an increase in email acquisition because readers will be more drawn to the topic of school choice.

Design

C: School Choice
T1: Border Crisis

Results

 Treatment NameConv. RateRelative DifferenceConfidenceAverage Gift
C: School Choice 1.8%
T1: Border Crisis 3.6%98.8% 99.9%

This experiment has a required sample size of 625 in order to be valid. Since the experiment had a total sample size of 3,286, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 98.8% increase in conversion rate
× 0% increase in average gift

Key Learnings

The Dear Reader pointing readers to our School Choice survey did not perform as well as its counterpart – the State of Our State survey. In fact, the School Choice variant produced a 42.3% decrease in email conversion rate with a valid 99.9% level of confidence.

The results give us valid proof that more page visitors were interested in offering their opinions on the general well being of our state covering a number of topics compared to those who wanted to offer their opinions on a specific topic. Is this evidence that we should boast more general surveys over single-issue surveys moving forward? Not necessarily, as it’s possible that school choice simply wasn’t a topic that is affecting that many people right now, or that people care deeply about.

The value proposition for each Dear Reader followed a similar template, so without a different approach in value proposition, the results really do boil down to the topic presented.

For our next test, we will swap the School Choice survey for the Illinois Forward survey, which is about the state’s fiscal standing and overall responsibility. We’ll found out if that topic outduels the State of Our State survey.

It’s important to test different dear readers to see which resonates best with our readers and deserves the long standing spot for our standard policy research dear readers. We will continue to test to determine which one has the greatest potential to rack up emails and enhance engagement.


Experiment Documented by NextAfter

Question about experiment #152110

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.