How to effectively communicate with respondents: principles of survey design

Abstract

This chapter provides an overview of the fundamentals of survey design and question construction for any researcher planning to implement an educational survey. Once a researcher establishes the purpose and goals of their research agenda, the next step is to construct a survey that can produce reliable and consistent data points for the study’s target population. For this to be successful, researchers and survey designers must understand the theories that guide survey design, and the impact that survey design has on data quality. While the applicability of this chapter spans all levels of education and research endeavors, the chapter integrates practical insights from large-scale, cross-sectional and longitudinal postsecondary studies. First, readers are introduced to main theories explaining how respondents interact with surveys and respond to questions. Two of the theories discussed in this chapter are the survey response process (the psychology on how respondents answer surveys—Tourangeau, Rips, and Rasinski, 2000) and the logic of conversation (the psychology on what information should be provided in surveys—Grice, 1975; Schwarz, 1996). The survey response process models a cognitive pattern of how respondents answer survey questions. Tourangeau and colleagues argue that there are four stages: comprehension, retrieval, judgment, and response. Knowing that respondents engage in each stage can help researchers understand how to design questions. The logic of conversation is a theoretical framework for understanding how people interact in conversations. When designing surveys, researchers can use this framework for understanding how their respondents will interact with the survey, and therefore what information should be included in the survey. This chapter examines the details of these theories and provides examples on how to improve the respondent’s survey taking experience. Second, this chapter will combine these and other prominent theories—such as the Gestalt Principles for visual design—to inform the reader of the best practices for survey design. The reader is provided an overview of the most critical components of question construction, including standards of question wording with respect to target populations, choosing appropriate question formats, essentials of logical routing and “looping” (e.g. administering the same set of questions for multiple instances), obtaining sensitive information while avoiding social desirability bias, and factors to consider across different modes of administration. Additionally, the chapter will discuss other concerns related to the survey design such as the potential use of alternate sources of data, basics of visual design (e.g., use of color and white space), importance of question order, and modes for administering the survey including the importance of mobile-friendly optimization. Results from experiments conducted during the 2015-16 National Postsecondary Student Aid Study field test interview, the 2012/17 Beginning Postsecondary Students Longitudinal Study pilot interview, and the 2008/18 Baccalaureate & Beyond Longitudinal Study field test interview are presented as examples throughout to help illustrate the importance of question wording and design choices. The chapter concludes with a summarization of the guiding theories and principles of survey design, and a consolidated list of best practices when designing survey questions for educational studies.