We know that poorly written survey questions can lead to inaccurate data, cause survey fatigue, and result in non-response.
Question makeover #1
There are two significant problems with this question. Let’s start with the question stem. Here, our first instinct is to answer in our heads “Well I am NOW!” We know they meant “before this moment, were you aware…” but unfortunately, the question wasn’t composed that way.
The second problem lies in the response options. They clearly wanted a 5-point scale (that every survey question should feature a 5-point scale or that five is necessarily the best number of scale points are pieces of bad survey advice we encounter quite frequently). But, does awareness really come in degrees? Can you be a little aware or very aware of something? And what is the difference between “a little aware’ and “somewhat aware” or between “very aware” and “extremely aware?”
We were so pleased to see this revision of that same question.
Here, the question stem is much more clear, asking whether respondents were “previously aware…” The construct “awareness” is now presented as dichotomous – something that requires a simple YES/NO or similar binary response. Another advantage of refining the question in this way is that they are easily able to measure 4 items in one brief, easy-to-respond-to matrix.
Question makeover #2
In this example, they tried to force all items onto the same scale, and as a result, the way these questions are phrased doesn’t quite make sense. “Were you able to…” implies an answer of yes or no, not a rating of excellent, good, etc. In addition, (and you probably caught this too!), this question is double-barreled. It’s asking two questions at once – whether the appointment process was quick AND easy. It’s entirely possible the process was easy to navigate, but not quick.
Again, we were so pleased to see an updated version of this question.
In this version they split up the two constructs – ability to schedule an appointment and ease of scheduling an appointment. Presenting them in phrases leads more naturally to a rating response though they also could have used a complete sentence for the stem too: “Please rate your ability to schedule an appointment.”
Lessons Learned
What can we learn from these makeover examples? Let’s return to where we started: Words matter. The way in which we ask questions matters – to respondents, and to us as survey researchers. Let’s ask clear, precise questions that align with our purpose for using a survey and what we really need to measure. And let’s take inspiration from these examples that it is worthwhile to improve survey questions when they aren’t working well.
