After facilitating a particularly fun abbreviated session of our Designing Quality Survey Questions course with Oregon Community Foundation staff earlier this year, I had a few colleagues approach me separately but with the same “aha” moment:

Much of the guidance provided about survey question design applies to other scenarios too, both in and beyond research! 

I was, of course, thrilled they had made the connection so readily to how to apply their learning from the course not only to surveys but their other work – grant application and reports, interviews, and planning meetings, and more. So much of our work as humans is inquiry, whether we label it that way explicitly or not.

Of course, the way we ask questions matters. This seems obvious, but posing the right question, in the right way, takes intention. 

A common, relatively simple example we see in survey design, and in many other circumstances, is when a questions is asked as though it is open-ended, but is actually a yes/no question, or invites a very specific kind of response, and isn’t a true invitation for sharing, reflection, or the kind of open feedback we really want and need from an open-ended question. There is a big difference between “Did the topic resonate with you,” and “to what extent did the topic resonate with you and  “in what ways did the topic resonate with you?” We’re going to get different answers to each of these questions. 

Here are a few of my favorite bits of guidance from survey design that can help us ask better questions in other scenarios. Big shout out to Jara Dean-Coffey and the rest of the Equitable Evaluation Initiative (EEI) team for some of this language, and their support for my practice of the Equitable Evaluation Framework (™) which has deeply influenced the way I think about this work. 

Name and clarify purpose

The clearer we can be about why we are doing something – whatever that thing is, the more easily we’ll be able to shape questions that help us reach our goals. Purpose-setting can look like lots of different things – goal-setting, developing research or evaluation questions, establishing learning objectives, etc. 

When designing a survey I usually work collaboratively with whomever needs or is going to use survey results to write a purpose statement and overarching research questions. Then we can use that as a reference point throughout the survey development (and analysis) process to make sure we stay on track (or at least are intentional about any evolution). We can do this, or something very similar for any scenario in which we are asking questions to learn something, we just have to ask ourselves: What are we trying to learn? Why? To what end?

What matters most in my experience is that we take the time needed to establish our purpose, we make sure there is shared understanding of that purpose among key partners and that we’ve documented this somehow so that we can refer back to the purpose whenever we feel ourselves drifting or unsure about next steps. Definitely don’t skip this step, and invest the time needed to do it well.

Get to know your ‘respondents’

It is tough to formulate a question for someone else without knowing at least a bit about them and the context from which they will answer. The more we can get to know whomever it is we are asking questions of, the more we’ll be able to craft questions that they can authentically answer. In Designing Quality Survey Questions, Sheila and I talk about using design thinking concepts to get to know intended survey design respondents. This applies to other scenarios too. How can we use the phases of empathizing and understanding to make our questions more meaningful to those who will answer them?

In survey work that might involve co-crafting questions with a group of potential respondents or those who know them better than we do as survey researchers or using a pretesting strategy like cognitive interviewing to better understand what respondents are thinking when they answer draft questions. In other efforts, similar strategies can be used to center our efforts around what will make sense to those we hope to engage with, and even to center their learning interests and needs. 

Sheila and I also talk about surveys as conversational. The more we can think about the act of asking questions as a part of being in a conversation (ideally as part of a broader or ongoing relationship) with whomever we are asking, the more we remember that the folks on the answering side of our questions are humans, and keep that at the forefront of our minds when we frame our questions.

Keep questions clear and easy to answer

Again, I know this seems like really simplistic advice, but unless we are thoughtful about why we’re asking a question, and we have some understanding of who will be answering it, it is tough to ensure that the question is clear and easy to answer. That said, we can do some simple things, building on that earlier advice, to help. 

I’ve lost track of the number of times I’ve told myself or others to “write questions that sound like a human wrote them.” We spend far too much time mired in the jargon and technicalities of our various fields and forget to take the time to find a way to ask our questions in plain, direct but kind, language. Doing so will literally lighten someone’s literal cognitive load, and likely make our questions more enjoyable to answer. 

I’m quick to admit that I stack questions and add more context than needed at times. I’m human and usually trying to do too much in any given information gathering scenario or conversation. But I don’t do this when I have taken the time to really think it through. How would I want to be asked about this? What am I really trying to learn here? How can I invite the person I’m asking this question to respond authentically? As Marcia Cone’ and the rest of the EEI team say “clarity is kindness.”

Minimize ‘respondent’ burden 

Fortunately, if you’ve named and clarified purpose, are centering your respondents, and are working toward clear, easy to answer questions, you’re already well on your way to minimizing burden on those you’re asking questions of. But it’s worth giving this bit of guidance a little more attention too. 

In survey design, we consider the cognitive load of individual questions or an entire survey tool, and aim to make the survey response experience as engaging as possible. We make sure we don’t already have the information we seek (and we’re asking just to confirm something we already know), or that it isn’t already available elsewhere. These are very reasonable things to think about in any other inquiry scenario. How much are we really asking of people? How long will it take? How can we shift to more respectful, or reciprocal work, rather than purely extractive? 

Whether a survey or any other scenario where we are asking questions and expecting answers that are useful in return, we can and should compensate people for their time and wisdom. We can also ensure that ownership of the information stays with those who shared it in the first place by sharing data back and supporting use of that information for those who engaged with us (in the ways they determine and desire, not the ways we might impose). 

There is no “right” way to ask any given question 

You might notice that I’m not offering or promising a formula here. There is no “right” way to ask any given question, exactly. Because the language we need, the approach we take, the answer options that we offer (when appropriate), always depends on the context that we and our respondents, or participants or co-conspirators sit in.

What other bits of survey design advice do you consider as valuable in other inquiry scenarios?

In future posts I hope to continue this conversation, bridging more survey design advice to other scenarios and sharing more examples.