How communal language impacted donation page visitors at year-end
Dallas Theological Seminary
The DTS mission is, “to glorify God by equipping godly servant-leaders for the proclamation of His Word and the building up of the body of Christ worldwide.” They strive to help men and women fulfill the Great Commission and the Great Commandment, or more simply: Teach Truth. Love Well.
Experiment Summary
Ended On: 01/28/2021
DTS had a sticky bar running on their site at year-end with language about a deadline to reach a $500,000 Challenge Gift. We wanted to test a hypothesis that adding softer, more relationship-based language in place of the transactional, deadline focused language would drive additional traffic to the donation page.
Research Question
We believe that using relationship-based language for site visitors will achieve greater donation page traffic.
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.60% | ||
| T1: | Treatment #1 | 0.24% | -60.0% | 96.5% |
This experiment has a required sample size of 2,461 in order to be valid. Since the experiment had a total sample size of 5,746, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
60.0% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
What we found after running the experiment was the treatment copy decreased conversions by 60%. For future experiments, we would want to match the value proposition more closely on the treatment in order to truly isolate the deadline and transactional Challenge Gift language from the communal language. For example, a gift to “reach lives with the truth of the Bible” could have been perceived as more motivating than “reach the world with God’s Word” so the results are too convoluted to draw a true conclusion. We would also like to run a similar test during a different season to see if this alters results at all since we know that donors are typically highly motivated to give during the month of December.