Purpose of Report

Given that health-system decision making would benefit from both traditional systematic reviews and health-system-specific data, this report investigates when and how to use primary data from health systems in real time with systematic reviews and articulates a framework for using health system data with systematic reviews to support health system decision making.

Key Messages

Based on our review of examples and methodologic guidance, as well as our experience conducting systematic reviews for various stakeholders, we recommend five basic principles regarding when and how to use unpublished health system data alongside of systematically reviewed data.

• Explicitly state the rationale for using unpublished data (i.e., to improve the strength and applicability of evidence, and/or to inform its implementation).
• Describe the details of the data source being used and why it was chosen (e.g. how relevant are the data).
• Characterize the limitations and biases of any included data through formal critical appraisal and if possible, working with a health system's QI and information systems staff and health system researchers to understand data and information-quality limitations.
• Specify how the findings from unpublished data support, refute, and/or otherwise add to findings from published data. If the unpublished evidence conflicts with the review's conclusions, discuss possible reasons for the discrepancy.
• Consider working in close partnership with health systems, which ideally includes a range of individuals such as clinical leaders and decision makers as well as QI staff and health system researchers.

Abstract

Systematic reviews are an important and necessary source of information to improve healthcare delivery; however, reviews of the existing research are often insufficient to address the decision-making needs of health systems. Incorporating data from health systems into traditional systematic reviews may be one way to improve their utility. In this paper, we map out ways in which health system data can be used with systematic reviews, articulate the scenarios for when health system data may be most helpful to use alongside systematic reviews (i.e., to improve the strength of evidence, to improve the applicability of evidence, and to improve the implementation of evidence), and discuss the importance of framing the limitations and considerations when using unpublished health system data in reviews (i.e., critical appraisal to understand the study design biases as well as limitations in information and data quality). To develop this framework, we used examples identified through literature searches and affiliations with four health systems that have the ability to use both internal and external evidence to support their clinical operations. Finally, we also offer recommendations to systematic reviewers who choose to integrate health system data and possible next steps in developing processes and capacity to routinely conduct this type of work.

Journal Citation

Lin JS, Murad MH, Leas B, et al. A Narrative Review and Proposed Framework for Using Health System Data with Systematic Reviews to Support Decision-making. Journal of General Internal Medicine. 2020 1 April. DOI: 10.1007/s11606-020-05783-5.

Citation

Suggested citation: Lin JS, Murad MH, Leas B, Treadwell JR, Chou R, Ivlev I, Kansagara D. Integrating Health System Data with Systematic Reviews: A Framework for When and How Unpublished Health System Data Can Be Used with Systematic Reviews to Support Health System Decision Making. Methods Research Report. (Prepared by the Kaiser Permanente Research Affiliates, Mayo Clinic, ECRI Institute-Penn Medicine, and Pacific Northwest Evidence-based Practice Centers under Contract Nos. 290-2015-00007-I, 290-2015-00013, 290- 2015-00005-I, 290-2015-00009-I). AHRQ Publication No. 19(20)-EHC023-EF. Rockville, MD: Agency for Healthcare Research and Quality. April 2020. Posted final reports are located on the Effective Health Care Program search page. DOI: 10.23970/AHRQEPCMETHQUALIMPRINTEGRATING.