RTI uses cookies to offer you the best experience online. By clicking “accept” on this website, you opt in and you agree to the use of cookies. If you would like to know more about how RTI uses cookies and how to manage them please view our Privacy Policy here. You can “opt out” or change your mind by visiting: http://optout.aboutads.info/. Click “accept” to agree.
Nuanced interpretations of statistically nonsignificant results were rare in Cochrane reviews - A metaepidemiological study
Gartlehner, G., Persad, E., Ledinger, D., Chapman, A., Gadinger, A., Feyertag, J., & Emprechtinger, R. (2023). Beyond statistical significance: Nuanced interpretations of statistically nonsignificant results were rare in Cochrane reviews - A metaepidemiological study. Journal of Clinical Epidemiology, 160, 46-53. Advance online publication. https://doi.org/10.1016/j.jclinepi.2023.06.007
OBJECTIVES: To assess the language used by systematic review authors to emphasize that statistically nonsignificant results show meaningful differences. To determine whether the magnitude of these treatment effects was distinct from nonsignificant results that authors interpreted as not different.
STUDY DESIGN AND SETTING: We screened Cochrane reviews published between 2017 and 2022 for statistically nonsignificant effect estimates that authors presented as meaningful differences. We classified interpretations qualitatively and assessed them quantitatively by calculating the areas under the curve of the portions of confidence intervals exceeding the null or a minimal important difference, indicating one intervention's greater effect.
RESULTS: In 2,337 reviews, we detected 139 cases where authors emphasized meaningful differences in nonsignificant results. Authors commonly used qualifying words to express uncertainty (66.9%). Sometimes (26.6%), they made absolute claims about one intervention's greater benefit or harm without acknowledging statistical uncertainty. The areas under the curve analyses indicated that some authors may overstate the importance of nonsignificant differences, whereas others may overlook meaningful differences in nonsignificant effect estimates.
CONCLUSION: Nuanced interpretations of statistically nonsignificant results were rare in Cochrane reviews. Our study highlights the need for a more nuanced approach by systematic review authors when interpreting statistically nonsignificant effect estimates.
