Statistics Research » Statistical Design & Analysis

Psychometrics

Whether assessing the performance of an existing scale or developing a new scale or test, our psychometricians bring the best modern measurement techniques to bear on such tasks. Careful design and assessment of scales assure survey administrators that their inferences from the data will be as accurate, reliable, and valid as possible. Psychometric techniques provide powerful tools that can distinguish useful from less useful measurement, and provide feedback on how successful are subsequent changes in items.

In today's statistical modeling, measurement models are also often directly incorporated into analysis and inference, using techniques such as structural equation modeling, in order to assure a proper accounting of all types of error (measurement and structural) found in an analytic model.

Focus Areas

  • Scale and test development and validation
  • Improvement on existing instruments that are imprecise, unreliable, or invalid
  • Incorporating measurement models in analysis for correct inference
  • Understanding sources of measurement error

Methodologies/Techniques

  • Item response theory
  • Factor analysis
  • Structural equation modeling
  • Latent class analysis
  • Generalizability theory
  • Classical test theory (reliability, validity, sensitivity, measurement error)

Capabilities

  • Assess item or scale reliability, validity, and sensitivity
  • Evaluate the quality of measurements found in scales used in surveys/questionnaires
  • Identify ways to reconstruct existing scales to give them greater precision, reliability, validity, and sensitivity
  • Optimally develop novel measurement scales
  • Appraise changes in scale structure derived from new populations
  • Tailor measurement instruments to special sub-populations
  • Assure meaningful measurement for all levels of a given measurement dimension
  • Sample-free scaling and scoring using methods based in item response theory (IRT)
  • Ascertain and apportion sources of measurement error across different measurements or raters (judges) using generalizability theory
  • Conduct analyses that include measurement models directly in the model to correctly include the inherent measurement error in scales

Applications

  • Educational outcomes
  • Health outcomes
  • Occupational assessments
  • Opinion surveys
  • Employee surveys

Projects

Health

  • Developing Patient Reported Outcome Measures for Complementary and Alternative Therapies. Developing a new measurement instrument for use in survey and outcomes research related to complementary and alternative medicine (CAM) evaluating several subscales through the use reliability statistics and factor analysis.
  • Optimization of a Scoring System for Near-Miss Maternal Morbidity. This RTI study attempts to validate previously published scoring measure using a perinatal database developed at a major academic medical center in Washington, DC., testing for equivalent sensitivity, specificity, and ROC curves, and exploring ways the measure could be improved.
  • Consumer Assessment of Health Plans Study. Psychometric work on the project included the design, development, and testing of surveys including several scales included within them and providing to lay and technical audiences useful and scientifically rigorous information about the reliability and validity of the surveys.
  • Data Coordinating Center for the NIH-DC Initiative to Reduce Infant Mortality. As part of the project team, our psychometricians provided psychometric guidance to the Data Coordinating Center staff, including modeling responses using item response theory (IRT), structural equation modeling, latent class analysis, and factor analysis.
  • Dynamics and Meaning of Adult Unintended Pregnancy. Guided the development of a new measure of women's intention with regard to pregnancy, to optimize the measure's psychometric properties.
  • Psychometric Analysis of Patient Satisfaction Questionnaire. Performed confirmatory factor analysis testing whether the response structure to a patient satisfaction questionnaire remained consistent across different antibiotic delivery modes.

Education

  • Preschool Curriculum Evaluation Research. Evaluation project including factor analysis of outcomes to develop composites and reliability checks on all scales used in project.
  • NICHD Study of Early Child Care. Various psychometric tasks on various projects including IRT analysis of a cognitive measure, various confirmatory and exploratory factor analyses of items, and many statistical models that included measurement models within them.
  • School Health Policies and Programs Study. Developed and implemented the assessment of the reliability and validity of data collected in the 2000 SHPPS.

Employment/Occupations

  • SIMPACTTM. The SIMPACT system (focusing on employee satisfaction, impact, and actionable results) is a work/nonwork organizational assessment and decision support tool that identifies life needs most strongly related to employee retention through predictive analytics. Sample-specific factor analytic and structural equation model analyses determined needed modifications to the work/nonwork model.
  • Occupational Information Network (O*NET) Data Collection. Operating through a consortium of the North Carolina Employment Security Commission and the Ohio Bureau of Employment Services, under a grant from the U.S. Department of Labor, the Occupational Information we conducted and reported on two separate psychometric analyses: a generalizability study and an assessment of data reliability.
  • Analysis of Employee Satisfaction Survey. Performed exploratory and confirmatory factor analysis to explore and confirm the structure of a web-based employee satisfaction survey administered to all employees leaving this major company.

Military

  • Unit Level Influences on Alcohol and Tobacco Use. This project is a component of the DoD Lifestyle Assessment Program (DLAP) with the goal of understanding which factors influence the use of alcohol and tobacco among high risk military personnel. Psychometric analyses (i.e. reliability, Rasch methodology, and factor analysis) evaluated several subscales within the Unit Level Survey.
  • Psychometric Evaluation of Navy-Wide Personnel Surveys. The purpose of this project was to conduct psychometric analyses of the Navy Personnel Survey, using techniques such as Item Response Theory and Structural Equation Modeling. Specific steps included: (1) evaluating the psychometric properties of current existing scales, (2), providing recommendations on ways to improve these scales and measures, (3) developing an index of Navy Climate, and (4) analyzing the relationship between scales and measures collected on the survey with military outcome measures.
  • An Examination of the Relationships Between Quality of Life (QOL) and Retention Behavior Among U.S. Navy Personnel. Extend previous research on QOL with Navy personnel by exploring the relationship between previously tested models and actual retention behavior. Structural equation modeling tested substantive hypotheses on QOL and reenlistment decisions. Rasch methodology evaluated new subscales on the survey.
  • Naval Health Research Center Task Order: Mental Health Issues Among Deployed Personnel. Assessed the structure and performance of numerous scales in this special population compared to the general population.