Getting better data from newsletter surveys

Recently, I had a great email conversation with Ted Fickes about surveys.
​
Ted runs Bright+3 and writes Future Community, a newsletter I love reading. He’s also a self-described survey nerd.

I like thinking about and designing surveys as conversations – an opportunity to engage, listen, and build a relationship. Of course, you need a strategy for reflecting what you hear back to people.

Ted Ficks, Bright+3

I agree with that approach wholeheartedly.

However, I’m not a fan of surveys in my day-to-day work, because I’m not trained as a survey designer. Survey design is a skill. I don’t have it, and I have other skills I’d rather learn. Like sewing.

However, I know surveys are the go-to method for gathering feedback from your audiences. I see no evil in surveys, but it’s important to know their limitations. For example, bad data is worse than no data. Surveys are easy to scale, but we don’t want to scale bad data, right?

How can we create surveys that give us more reliable information we can use?

Fan Friction

There are two most common problems I see with most surveys:

1. They are too long.
2. They are too difficult.

These two things create too much friction. Friction leads people to close the window and move on to something that isn’t adding more work to their day.

Five minutes is a long time to fiddle with a survey. Recently I’ve seen surveys go even longer. 10 minutes. 12 minutes! I can listen to “Marquee Moon” in less than 12 minutes!

This severely limits your responses, too. If you are okay with fewer responses, why not meet people in a coffee shop or park to talk for 15 minutes? You’ll have a better discussion and get better insights.

Secondly, the questions are too difficult. People are notoriously bad at accurately describing their behavior. This is even more true for hypothetical future behavior. Part of this that is our memories aren’t as good as we think. Plus, we care about how we appear to other people.

So when you ask “If we were to offer more newsletter options, which would you subscribe to,” you have two obstacles:

1. Most people don’t know what they would do in the future.
2. They don’t want to hurt your feelings.

This typically leads people to answer, “sure why not.” But when your survey results always say people will subscribe to more offerings, does that mean there is this an endless fountain of interest — or that it’s not the right question?

How to simplify your newsletter survey

My recommendation is to ask simple questions, in plain language, so that people can answer quickly and truthfully. Most of these have an immediate correct answer, and Yes/No is a great way to get there. For example:

What’s your zip code?
Tons of uses from identifying coverage gaps to pitching advertisers.

Have you downloaded our app?
Could be a useful way of measuring awareness, possibly across channels.

What neighborhood do you live in?
​This could be super useful for understanding how people define where they live compared to their zip code.

How long have you lived in this city/state/region?
​Newer residents have different needs — and gripes — than long-time residents

Did you ride public transit this week?
How do your readers experience the city?

Not that all of these questions are past tense. They are all questions that are easy to answer truthfully without thinking too much. This means you are less likely to collect bad data.

These questions are also minimally invasive, so the stakes are pretty low for readers. When you start asking age and income questions, it gets personal. People will start dropping off.

…but don’t obfuscate

Simple is great, but oversimplification often leads to bullshit dressed up as rigor.

Like Net Promoter Score. NPS is that “How likely are you to recommend our company to a friend or colleague” question you see all the time.

The fact is, NPS is a garbage number that’s easy to put in place and tells you nothing.

Want me to prove it? Think about it for 30 seconds. What’s the difference between any two numbers on the scale? What makes an experience a 7 and not an 8? Now how does that work between different people? Are my 9 and your 9 the same? Of course not.

People smarter than me have studied that NPS fails at the one thing it claims: measuring the propensity for more purchases and referrals.

We find no support for the claim that Net Promoter is the “single most reliable indicator of a company’s ability to grow.”​

~ Keiningham, et al.

Also, MBA Geniuses love NPS, which is plenty enough reason to avoid it.

One tip for asking harder questions

I know, The Boss is going to want to see certain questions on your survey. So if you must include questions that will take more thought, help the reader get some momentum so they don’t bail. Ask the quick & easy questions up front, so that there are only 1-2 harder questions at the end.

This way, if they bail, you’re more likely to get something rather than nothing.

Finally, think about asking fewer questions more frequently. Instead of one five-minute survey, why not one survey every other month that takes less than 60 seconds to complete?
Bonus link

• ​On Surveys by Erika Hall. Erika is a big influence on my work, and her books are fantastic.

===

[bloops]

💬 Andrea Grimes surveyed 491 writers and 2,200+ bylines for the 3rd annual Deep Dive into Texas Magazines.

💬 The smartest thing I’ve seen in 2024 are these women’s hockey jerseys.

Commitments to 99 Newsletter Project readers

The 99 Newsletter Project does not use AI or LLMs to generate any part of this newsletter or 99NewsletterProject.com. I’m a strong believer that I won’t ask you to read something that no one bothered to write.
​
The 99 Newsletter Project does not use affiliate links or accept sponsored content. Click on anything! I’m not getting paid to sell you products I don’t make.