9 Reflective Questions You Should be Asking as You Design a Survey

Kim here, back writing with my favorite survey design partner and co-author Sheila Beth Robinson.

Questions are a powerful tool no matter what work you’re doing. Questions can catalyze thinking and shift perspectives. In designing surveys – whether for our own projects or for clients – we ask a series of questions we now know help us develop a high quality instrument, capable of capturing the rich, nuanced data we need. Some of these questions are for the planning and pre-drafting stage and help us resist the temptation to jump straight into question drafting. Others are for the question development stage.

Here are the (interrelated) questions we ask during survey development and how to use them:

1. What is the purpose of the survey? Or, what are your main goals in conducting the survey?

Get clarity about your survey purpose and goals. This will help set you on the right path to developing questions, help you refine (and trim) questions to keep your survey as brief as possible, and communicate clearly to respondents about why they should respond. Write a brief list or short paragraph stating your purpose and goals, and use this as a reference throughout the process.

2. What are the big questions the survey will help answer? 

Do you have one or more broad, overarching research or evaluation questions? If not, try to identify your information needs and formulate a few key questions that data from this survey will help answer. These broad questions can help in the same way as your survey purpose – helping you to maintain course and make smart decisions throughout the survey development process.

3. Is a survey the right tool given your overarching goals or purpose? Why?

Think critically about this. We too often jump to surveys as our data collection tool of choice, and they may not be the right one! Clarifying purpose, goals, and identifying the big questions you are trying to answer can help confirm, or redirect, you in choosing to collect data via a survey. Surveys are often not the right choice if the answers to your big questions need to be detailed, nuanced, or richly descriptive, for example. Instead, you might realize that a set of interviews or focus groups would be a better approach. 

Surveys are useful when we need to gather data from a lot of people, or to offer a large number of people an opportunity to offer their perspectives on a topic. They also work well when we have really specific questions to ask, and we can anticipate potential responses (so that we can create specific response options). 

Feasibility is also important to consider – both in terms of time and budget. Surveys aren’t always the fastest option, or the simplest when it comes to analysis, but they can be efficient if you have a clear purpose, a well designed survey, and a smart communications plan. 

4. Who are you trying to reach? Or, who are your potential or desired respondents?

The more you can articulate who you want to hear from and why, the easier it will be to plan for communicating with them about the survey (and why they should respond). 

5. What do you know about your desired respondents? 

The more you can learn about your desired respondents, the more you can ensure your survey is easy for them to respond to, or even enjoyable. Centering respondents’ perspectives can help you design questions that are clear and meaningful to those you hope will answer them, which can result in better data.

Exploring existing data available about desired respondents can also reveal information you don’t need to collect! Sourcing data elsewhere can save you and your respondents time and headaches. 

6. How can you best encourage response?

Knowing as much as possible about your respondents can help you determine why, or under what conditions they might be willing/able to answer your questions. Think about whether more informal or formal communications methods or approaches would work best, or whether you need help encouraging respondents from someone they know and trust. Or, whether offering an incentive is an ethical, reasonable, and feasible choice. 

7. How does this question (or set of questions) serve the survey purpose or goals? How does it align with your big questions?

Only questions that clearly serve the survey purpose and goals, and can be mapped to the big questions should be included. If you don’t need a question, don’t ask it. 

8. Is the question (or set of questions) clear and easy to answer? 

Survey questions that are clear and easy to answer are written in plain language, free of jargon or unnecessarily complicated words or phrases. They will make sense to your respondents, given what you know about them, their experiences, backgrounds, etc. Closed-ended questions will have response options that match the question stem (the language is aligned), are mutually exclusive, and are inclusive (respondents will be able to find an answer choice that matches their experience). 

It should be immediately obvious to the respondent how to answer the question – what is being asked of them. This can mean including brief instructions like “select the best fit” or “choose all that apply” alongside the question itself. 

9. And finally, are you minimizing respondent burden as much as possible? 

If you can clarify question language, or eliminate questions to shorten the length of your survey, do that. Anything you can do to minimize the burden placed on respondents – either in trying to reduce the demand for cognitive effort by using plain language and clear instructions, or in making the survey accessible, engaging, and easy to complete will help your response rate (and is simply kinder to your respondents).  

What are we missing? 

Are there other questions you ask yourself or your clients when developing a survey? We would love to add to our list. 

You can also read more about our approach, and more advice about survey question development in our book, Designing Quality Survey Questions. Here’s an excerpt:

We take to heart the notion that a survey is a form of conversation and social exchange and that respondents are rarely, if ever, compelled to answer questions for us. This drives us to use a “respondent-centered” design process. Taking respondent needs, interests, and abilities into account allows us to maintain integrity as researchers; we respect and care for our respondents’ well-being and avoid burdening them with unnecessary or unduly difficult questions. Surveys must be more than a quick list of everything a researcher would like to know about respondents dashed off in a series of questions. Instead, we think of surveys as conversations built on a foundation of shared language. To frame that conversation such that we are able to gather the needed information, we must thoughtfully employ a purposeful survey design process that takes into account an understanding of and empathy for our potential respondents (Robinson & Leonard, 2019, p. 3)

Or, you can contact us about a survey design workshop for your organization!

Photo by Celpax on Unsplash