Matrix questionnaire design to reduce measurement error

Abstract

The full set of questions may be essential to fulfilling the survey objectives, yet long survey instruments can be taxing to respondents. As the respondent continues to answer survey questions for an extended period of time, they may not provide the same level of consideration to all survey questions – providing answers with greater measurement error towards the end of the survey. There is little empirical evidence to support the tradeoff between survey length and measurement error, leaving survey practitioners to rely on expectations to support improvement of survey estimates using a shorter survey. Furthermore, empirical research is in dire need of methods to address the challenge of collecting the full set of variables while minimizing the survey length. One potential solution is the use of split or matrix questionnaire design in which the survey is modularized and respondents are randomly assigned to receive a different subset of the survey modules. In this way, no questions are asked so late in the survey that the measurement error properties of the data are compromised. The full sets of variables are then multiply-imputed for all respondents. This method provides a tradeoff between measurement error in the long survey and increased variances of estimates in the matrix design. We pose two research questions:
1. Whether there is greater measurement error when questions are asked late in the survey, and 2. Whether a matrix questionnaire design can provide estimates with lower total error (MSE).
We use data from a web survey experiment in which a set of questions were asked either early or late in the instrument. We found strong evidence for greater measurement error when the questions were asked late in the survey. This paper will present an evaluation of the use of a matrix questionnaire design instead of the full survey.