A Roundup of Survey Design Resources

Kim here, writing with the magnificent Sheila B. Robinson of the Evaluspheric Perceptions blog.

Since agreeing that we would co-author a series of blog posts on surveys with a focus on composing good questions, we have discovered countless other blog posts, websites, journal articles, and books on survey research from a variety of fields and perspectives, many of which feature discussions of and advice on question construction. Of course, we have a few personal favorites and well dog-eared texts:

Sheila likes How to Conduct Surveys: A Step-by-Step Guide 3rd Ed. by Arlene Fink. Fink offers basic advice, such as “you have to define the attitude, belief, value or idea being measured” (p. 11) but little else on question wording. Like other texts, Fink addresses question types, response options, question order, etc. and facets of survey research.

Kim regularly consults Dillman’s Internet, Mail, and Mixed-Mode Surveys.  But Dillman doesn’t really get at the guts of our original question: “What makes a GOOD question GOOD?” Oh sure, there are threats to validity to consider, and reliability considerations, and some general advice that is as much common sense as it is good practice (e.g. don’t use double negatives). But little detail about what works in terms of question wording itself.

Unlike other evaluation areas we study, there does seem to be agreement about the architecture of effective questions. Or, at the very least, we’ve encountered plenty of “what NOT to do” advice: avoid jargon, leading questions, loaded questions, and double-barreled questions. What we did NOT find in the literature base, however, is specific advice on wording that would instill in a survey researcher a high level of confidence in a question’s ability to elicit the most honest, most accurate, most valid and reliable response. In other words, we found the science, but not the art.

Tom Archibald and Jane Buckley describe in an AEA365 post “The ‘je ne sais quoi’” of evaluative thinking: “an essential, yet elusive, ingredient” of evaluation capacity building and evaluation practice. This is how we feel about the art of writing good survey questions – it’s an essential, yet elusive ingredient of composing effective surveys.

Though we continue to search for all the details we seek, here are a few resources, some old and some new, that have stood out in our exploration (Note! This is hardly an exhaustive or representative roundup):

Earl Babbie’s classic tome, Survey Research Methods 2nd Ed., is still considered a seminal work on the topic. First published in 1973 and purportedly the first book on this topic written for undergraduates (although it’s appropriately rigorous for grad students and researchers), it is a compendium of not only survey research, but also social science research itself. In fact, Babbie barely mentions surveys for the first 50 pages. Instead, he outlines the logic of science before moving specifically into social science, and then finally into survey research design. He then sails through sampling, conceptualization and instrument design, and dispenses a very healthy dose of analysis and statistics.

Fowler tackles our core question, “What is a good question?”, in the introduction to Sage Applied Social Research Methods guide: Improving Survey Questions: Design and Evaluation and discusses question wording in a brief section of Chapter 4. Fowler states,  “A good question is one that produces answers that are reliable and valid measures of something we want to describe” and lists five characteristics of good questions: (1) questions are consistently understandable; both questions (2) and answers options (3) are consistently communicated; and that respondents are able (4) and willing (5)  to answer said questions. More of Fowler’s principles are likely to appear in future posts in this series (and to benefit our survey design practice!).

Several AEA365 posts cover the subject:

• Jan Losby and Anne Wetmore in “whether to survey or not to survey—that is the question” discuss the most important consideration of all — the survey purpose and ask evaluators to consider the why, what, who and how of the survey before embarking on conducting one. They also present a number of resources, including this gem from the American Association of Public Opinion Research.
• Jade Caines blogged on survey design as part of EdEval week. She too stresses the importance of “defining exactly what it is that is being measured” — the ‘what’ that can so easily be taken for granted.
• Heather Bennett writes on her key survey question lesson: “the quality of the question asked has the greatest impact on data analysis process.”

Qualtrics, the web-based survey giant, even composed a series of posts on survey design not a terribly long time ago.

And, we even discovered a few other attempts to summarize resources, as we are trying to do, such as on the Instigator blog.

There are so many more, though no one text that seems to completely answer our question about survey questions. Perhaps we’re missing something yet…. Regardless, we want to know:

What’s YOUR favorite survey design text?

Coming soon: In our next post in this series, we’ll summarize our favorite advice in the form of dos and don’ts for survey question design.