The art and science of asking questions is the source of all knowledge. – Thomas Berger

Hey readers, Kim here, writing once again with the marvelous Sheila Robinson, of the Evaluspheric Perceptions blog.

It’s survey design season, so get ready to flex those question design muscles! Well, to be truthful, it’s always survey design season in our data-saturated evidence-hungry society. As surveys have become ubiquitous, it is incumbent upon survey researchers to to cut through all the noise by developing the most effective instruments we can. And what’s the best way to get ready for any endeavor that requires flexing? A warm-up! Just as failure to warm up for physical activity can invite injury, diving into survey question design without a preparation process can introduce the possibility of gathering bad data. Remember the principle of GIGO?

Sheila and I have co-authored several posts on our shared passion for good survey design. You can find these posts here, here, here, and here.

For this post, we’ve created your warm up routine: questions we ask before developing any survey. Thinking through these will help you design the most effective instrument for getting a rich and robust dataset that will help answer key evaluation questions.*

Before finalizing any survey (ideally, before you begin to draft it!), you should be able to answer each of these questions. Though some questions may seem closely related, they can illuminate subtle nuances about information needs, and inform how best to craft a question that will elicit that information. We’ve organized these into loose WHAT?, WHY?, SO WHAT?, NOW WHAT? categories, followed by special considerations that connect the question design process to the analysis plan.

WHAT?

• What information do you need?
• What are you attempting to measure?
• What is MOST important to know about this?

WHY?

• Why is this important to know? (i.e. does it directly connect to an evaluation question?)
• Is the information that will result from this question necessary to inform a potential decision or action?

SO WHAT? NOW WHAT?

• What’s the ‘so what’ factor (i.e. so what if ‘x’ percent of respondents have engaged in ‘y’ behavior before; so what if ‘x’ number of respondents strongly agree to a statement)?
• What will you do with this information? (not just how will you work with the information, but what action(s) may be taken, how might it be used to make decisions, improve programs, etc.)?

Special considerations once questions are drafted:

• Are you prepared to analyze the results from this question (e.g. are you equipped to analyze a “check all that apply” question? A rank order question? A Likert (or Likert-like) question? An open-ended question)?
• Can you anticipate how a respondent from your population or sample might answer?
• Is this an easy question for respondents to answer? Will they be able to answer quickly or in a reasonable amount of time?
• If they do answer quickly, is it likely they will be able to provide rich, accurate, and meaningful data by answering this question?
• Are there some common answers that might make what might otherwise be an open-ended question a better multiple choice question with an “other” option?

Of course, strategies such as informally consulting colleagues, piloting the survey with potential respondents, or cognitive interviewing can help answer some of these questions too. Working through the questions in this post before engaging in these strategies, however, will elevate those efforts, make good use of people’s time, and even better ensure your survey will net the information you need.

Have we missed anything? What questions do you ask yourself (or others) during the survey planning and drafting process? Add yours in the comments!

*You have your evaluation questions, right? And, you must have determined that a survey is the best way to capture data from potential respondents to answer those questions, right? And, you’ve identified the purpose of the survey, right? If not, then take a BIG step back. You’ll thank us later….