How testing different Dear Reader survey topics impacted email acquisition
Illinois Policy Institute
Experiment Summary
Timeframe: 05/17/2023 - 06/10/2023
IPI has two active surveys: State of Our State and School Choice. We want to see which survey topic is most appealing to our reader audience and can collect the most emails, so we’ve separated them via a 50/50 Dear Reader experiment.
These Dear Readers are implemented at the end of policy research articles, so 50% of readers will be asked to complete the State of Our State Survey and 50% will be asked to complete the School Choice Survey once they are done reading their respective article.
Let’s see which version acquires the most emails.
Research Question
We believe that implementing a dear reader asking for opinions on the active topic of school choice in Illinois for page visitors will achieve an increase in email acquisition because readers will be more drawn to the topic of school choice.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | School Choice | 1.8% | |||
| T1: | Border Crisis | 3.6% | 98.8% | 99.9% |
This experiment has a required sample size of 625 in order to be valid. Since the experiment had a total sample size of 3,286, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 98.8% increase in conversion rate
× 0% increase in average gift
Key Learnings
The Dear Reader pointing readers to our School Choice survey did not perform as well as its counterpart – the State of Our State survey. In fact, the School Choice variant produced a 42.3% decrease in email conversion rate with a valid 99.9% level of confidence.
The results give us valid proof that more page visitors were interested in offering their opinions on the general well being of our state covering a number of topics compared to those who wanted to offer their opinions on a specific topic. Is this evidence that we should boast more general surveys over single-issue surveys moving forward? Not necessarily, as it’s possible that school choice simply wasn’t a topic that is affecting that many people right now, or that people care deeply about.
The value proposition for each Dear Reader followed a similar template, so without a different approach in value proposition, the results really do boil down to the topic presented.
For our next test, we will swap the School Choice survey for the Illinois Forward survey, which is about the state’s fiscal standing and overall responsibility. We’ll found out if that topic outduels the State of Our State survey.
It’s important to test different dear readers to see which resonates best with our readers and deserves the long standing spot for our standard policy research dear readers. We will continue to test to determine which one has the greatest potential to rack up emails and enhance engagement.