Research participant selection in non-English language questionnaire pretesting

Abstract

In survey research, cognitive interviewing has been widely used as a tool for pretesting to improve the quality of questionnaires. However, the cognitive interviewing method relies on interview participants' ability to articulate their thought process by answering the probe questions asked during the interview. Including participants who exhibit difficulty in understanding the probe question as intended by the researcher and difficulty in providing the appropriate answer pertinent to the probes affects the quality of cognitive interview outcome. Using main interview and its paradata from 258 non-English language (Chinese and Korean) cognitive interviews which pretest the American Community Survey questionnaire, this paper examines how levels of participant difficulty exhibited during the non-English cognitive interview affect the interview outcome. Furthermore, this paper provides practical guidelines about participant selection to balance interview quality and cost for pretesting translated survey questionnaires. We found that interviews with participants who exhibited greater difficulty were lengthier and did not add more value to the findings; in addition, those participants were less likely to complete any questionnaires in the future.