Now, THAT’s a very good question!
Kim here, writing with the fabulous Sheila B. Robinson of the Evaluspheric Perceptions blog.
We have been tweeting and emailing about challenges related to survey design and as a result, this is the first in a series of posts we have co-authored about our discussions.
I (Kim) have recently been flummoxed by the lack of good simple, current, published advice (preferably evidence-based) about phrasing survey questions. I feel like my go-to sources for survey design don’t include enough detailed information about what works when crafting questions.
We’ve shared a few resources with one another, and continue to digest those and seek others. There are certainly great resources covering threats to validity, reliability concerns, and broad advice like “use simple language”, “avoid double negatives”, etc. But nothing really tackles what we’ve been discussing — what makes a good question good – particularly in terms of phrasing? Is it hard to generalize advice about wording questions because context is too great a driver of that success? Are there known best practices regarding this aspect of survey design that we just haven’t discovered yet?
Despite a (at least perceived) lack of solid published information about survey question crafting, anecdotal advice abounds about how questions should be phrased. Some advice is hard to verify, or confirm. For example, take advice Kim has heard about not using emotive language (e.g. “do you feel included…”) because it’s too subjective. Yet, wording questions in other ways (e.g. “I am included”) seems in some ways more direct (which could be a good thing) and in other ways may become more difficult to answer. It’s certainly possible these two phrasings could result in different answers; two questions that seem to ask the same thing, may in fact, be about two different things.
Ultimately, it is MUCH easier to be critical of poorly crafted questions than than to write successful ones. As surveys have become ubiquitous, we’ve both encountered numerous examples of “bad” questions.
Consider this example: Sheila recently analyzed data from a brief survey of open-ended questions asked of a small number of practitioners across a few departments. The first question was “In what areas do you feel you have expertise?” The purpose of the question was unclear, and it could have been asked for any number of reasons. Was there intent to build a database of department members with expertise in different areas? Was there a need to get a general sense of departmental knowledge or experience? The word “expertise” is problematic as well, and open to interpretation. In fact, many respondents crossed out the word and substituted “experience” or “strength.” But what we really don’t know from a question worded in this way is whether people have simply had training or exposure in a certain areas, whether they have a certain level of confidence in a certain area, whether they feel they have enough knowledge or skill to practice in the area, or whether they have the ability to train others in that area. These are all vastly different questions, or at the very least one question stem – the “areas” with several response options (i.e. I have had training… I consider myself an expert, etc.). So, after analyzing data from this question with current wording, what do we really know now? The answer is… not very much.
As evaluators, we often think we have a good sense of how to effectively craft survey protocols and feel confident in our ability to ask good questions, but wouldn’t it be wonderful to have solid evidence for what works regarding survey phrasing? What resources would you would recommend that tackle this topic?
And even more interestingly, perhaps — What do YOU think it is that makes a good question good with regards to its phrasing?
Teaser! Look for future collaborative posts on: survey design resources, dos and don’ts of survey question design, and related topics.
Except for completely open-ended questions like “What didn’t we ask that you think we should know about this issue?” (and a willingness to consider whatever the response), I’m not a big fan of using open-ended questions on self-administered surveys. It’s hard enough to write close-ended questions in a manner in which everyone is answering the same question, even with inserting definitions, labeling all scale points, etc. Unless questions are about some technical, specialized or well-bounded topic, people will respond from the context of their unbounded human experience. Therein lies the problem. No “paper-and-pencil question” can begin to accommodate the myriad ways any human being will respond from their own complex, nuanced human experiences. Add to that what Shakespeare taught us: “There are more things in heaven and earth, Horatio,
than are dreamt of in your philosophy,” and it becomes a near impossible task for both questioner and respondent. As a qualitative researcher, you know where I’m going with this–the only way to ask open-ended questions is with in person interviews with individuals or groups. There you have the opportunity to clarify and customize the question to the respondent’s frame of reference, to probe for understanding, to uncover the meaning behind the words, and learn about what you may have never thought to ask.
Pingback:A Roundup of Survey Design Resources | actionable data
Pingback:Put me in, coach…I’m ready to survey! (cross post with Evaluspheric Perceptions) | actionable data
Pingback:Designing Effective Surveys Begins with the Questions BEFORE the Questions! | actionable data