Conveying results from qualitative comparative analyses within systematic reviews

Abstract

Research Objective
Qualitative comparative analysis (QCA) is a supplemental method proposed for synthesizing results from systematic reviews of complex interventions but is unfamiliar to most end users of systematic reviews. In our previous research, end users identified several barriers to understanding QCA results, including presentation formats. The objective of this study was to identify the most effective presentation approach to communicating results from a QCA within the context of a systematic review.

Study Design
Randomized experiment. We provided all participants with information about a hypothetical systematic review of a complex intervention for a common health condition then randomized each participant to review the results of the QCA in 1 of 3 presentation formats (text, table, or graphic). Each format contained the same results information. We then assessed ease of understanding (ie, subjective comprehension) and objective comprehension of the QCA results with an online questionnaire.

Population Studied
254 clinicians, public health practitioners, policy makers, and health researchers who had read at least 1 systematic review within the previous 5 years that we recruited via email from U.S.- and U.K.-based public health, health care, and academic organizations.

Principal Findings
We found a statistically significant difference (P = .035) across the 3 presentation formats in subjective comprehension as assessed by an index we created from 6 items (Cronbach’s alpha = 0.89). The mean subjective comprehension was highest for the graphic format (2.61 [SD 0.56]) followed by text (2.57 [SD 0.55]) and table (2.40 [SD 0.55])) on a scale of 1 (poor comprehension) to 4 (good comprehension). In preplanned comparisons, participants who were randomized to the graphic format scored higher compared to the table (P = .018), and participants allocated to text format scored higher compared to table (P = .034), but these differences were not considered statistically significant based on a Bonferroni-adjusted P value of .0167.

We assessed objective comprehension with 7 items designed to assess parameter interpretation and 7 items designed to assess configuration interpretation. The mean number of correct parameter items was 4.04 (SD 1.84), and the mean number of correct configuration items was 4.61 (SD 1.36). No significant differences across presentation formats were observed across groups with regard to objective comprehension overall (P = .071). However, we identified a statistically significant difference across groups for parameter interpretation (P = .001) but not for configuration interpretation (P = .090). For parameter interpretation, scores among participants that received the graphic format were significantly higher (4.70) than scores for participants who received the text (mean 3.88, P = .001) or table (mean 3.67, P = .004) formats.

Study limitations include the use of a convenience sample, evaluation of the different formats using only 1 example systematic review, and participants did not evaluate one format against another.

Conclusions
A table may be the least effective format for presenting results related to a QCA within a systematic review of a complex intervention. A graphic format appears to be superior on some measures of objective comprehension.

Implications for Policy or Practice
Researchers using QCA within a systematic review should be aware of differences in reader understanding based on presentation format and should consider communicating results with a graphic when possible.

Primary Funding Source
Agency for Healthcare Research and Quality.