1. What is the frequency and severity of medication errors and adverse drug events?
2. What patient safety measures or indicators have been used to examine the frequency and severity of medication errors and adverse drug events?
3. In what ways is computerized clinical decision support used for preventing medication errors and adverse drug events, and in what settings is it used?
4. What is the rationale for using computerized clinical decision support to prevent or mitigate the harms of medication errors or adverse drug events?
5. What are the effectiveness and unintended effects of using computerized clinical decision support on the frequency of medication errors and adverse drug events?
6. What are the most common barriers and facilitators of implementing computerized clinical decision support to reduce the frequency of medication errors and adverse drug events?
7. What resources (e.g., cost, staff, time) are required for implementation of computerized clinical decision support practices?
8. What toolkits are available to support implementation of computerized clinical decision support to reduce the frequency of medication errors and adverse drug events?

The Agency for Healthcare Research and Quality (AHRQ) Making Healthcare Safer (MHS) reports consolidate information for healthcare providers, health system administrators, researchers, and government agencies about patient safety practices (PSPs) that can improve patient safety across the healthcare system—from hospitals to primary care practices, long-term care facilities, and other healthcare settings. In Spring of 2023, AHRQ launched its fourth iteration of the Making Healthcare Safer Report (MHS IV).

Computerized clinical decision support (CDS) as a PSP was identified as high priority for inclusion in the MHS IV reports using a modified Delphi technique by a Technical Expert Panel (TEP) that met in December 2022. The TEP included 15 experts in patient safety with representatives of governmental agencies, healthcare stakeholders, clinical specialists, experts in patient safety issues, and a patient/consumer perspective. See the Making Healthcare Safer IV Prioritization Report for additional details.

Adverse drug events, defined by the Institute of Medicine as "an injury resulting from medical intervention related to a drug."^1,2 Adverse drug events are a leading type of healthcare-related harm, accounting for 39% of all adverse events in the inpatient setting.³ Of these events, it is estimated that 27% are preventable and 24% have a severity of serious (defined as causing "harm that resulted in substantial intervention or prolonged recovery") or higher. Similarly, these events are also common in the outpatient setting with an estimated 25% of patients experiencing an adverse drug event after an encounter. Of these events, 11% are preventable, 28% are ameliorable, and 13% are serious.⁴ There is substantial opportunity to reduce the frequency and severity of adverse drug events and improve patient outcomes.

Computerized clinical decision support (CDS) can provide medication-related alerts, such as reminders or recommendations, to prevent medication errors and ameliorate adverse drug events at the point of care. Making Health Care Safer (2001) addressed "Computerized Physician Order Entry (CPOE) with Clinical Decision Support Systems (CDSSs)" [Chapter 6] and summarized eight studies, as well as potential for harm, cost and implementation.⁵ The report concluded that clinical decision support systems improve the quality and safety of prescribing; however, information on the impact on patient outcomes was limited. MHS II (2013) provided an update focused on "Computerized Provider Order Entry With Clinical Decision Support Systems: Brief Update Review" [Chapter 41] and summarized three systematic reviews, unintended consequences, implementation and cost.⁶ This report stated that "conclusions regarding CPOE+CDSS in the 2001 edition of "Making Health Care Safer" thus appear to stand largely unchanged a decade later." MHS III (2020) did not address CPOE with CDSS. MHS III focused on other PSPs for "Reducing ADEs in Older Adults" [Chapter 9] and covered CDS for other health topics, but not adverse drug events.⁷ During the prioritization process, the MHS IV TEP noted that CDS is a very broad topic and would likely need narrowing to either specific outcomes or specific clinical contexts.

The overall purpose of this rapid review is to determine the effect of computerized clinical decision support on the key clinical outcomes of medication errors and adverse drug events. We will also consider unintended outcomes such as alert fatigue, low value alerts, and unsubstantiated overrides of alerts, and how computerized clinical decision support can be effectively implemented.

For this rapid review, strategic adjustments will be made to streamline traditional systematic review processes and deliver an evidence product in the allotted time. We will follow adjustments and streamlining processes proposed by the AHRQ Evidence-based Practice Center (EPC) Program. Adjustments include being as specific as possible about the questions, limiting the number of databases searched, modifying search strategies to focus on finding the most valuable studies (i.e., being flexible on sensitivity to increase the specificity of the search), and restricting the search to studies published since 2015, in English and performed in the United States, and having each study assessed by a single reviewer. Depending on the expected volume of literature, the EPC team may opt to have a randomly selected 10% sample of articles checked by a second reviewer or use the artificial intelligence (AI) feature of DistillerSR (AI Classifier Manager) as a second reviewer at the title and abstract screening stage, as described below in the section on Data Extraction.

We will search for recent high quality systematic reviews and will rely primarily on the content of any such systematic review that is found. We will not perform an independent assessment of original studies cited in any such systematic review.

For topics that focus on a PSP that may address a variety of different harms, we will answer Review Questions 1 and 2 by focusing on the harms and patient safety measures or indicators that are addressed in the studies we find for Review Question 5. For Review Question 2, we will focus on identifying relevant measures that are included in the Centers for Medicare & Medicaid Services (CMS) patient safety measures, AHRQ's Patient Safety Indicators, or the National Committee for Quality Assurance (NCQA) patient safety related measures.

We will ask our content experts to answer Review Questions 3 and 4 by citing selected references, including PSPs used and explanations of the rationale presented in the studies we find for Review Question 5.

For Review Questions 6 and 7, we will focus on the barriers, facilitators, and required resources reported in the studies we find for Review Question 5.

For Review Question 8, we will identify publicly available patient safety toolkits developed by AHRQ or other organizations that could help to support implementation of the PSPs. To accomplish that task, we will review AHRQ's Patient Safety Network (PSNet) and AHRQ's listing of patient safety related toolkits and we will include any toolkits mentioned in the studies we find for Review Question 5. We will identify toolkits without assessing or endorsing them.

We will search for original studies and systematic reviews on Review Question 5 according to the inclusion and exclusion criteria presented in Table 1.

Table 1. Inclusion and Exclusion Criteria

Our search strategy will focus on databases expected to have the highest yield of relevant studies, including PubMed and the Cochrane Library, since 2015, supplemented by a narrowly focused search for unpublished reports that are publicly available from governmental agencies or professional societies having a strong interest in CDS.

To efficiently identify studies that meet the eligibility criteria, we will distribute citations from the literature search to team members, with plans to have the title and abstract of each citation reviewed by a single team member. The team will decide whether it has enough time and resources to ask a second team member to check a 10% sample of citations to verify that important studies were not excluded after the review of titles and abstracts. Alternatively, the team may opt to use the DistillerSR AI Classifier Manager as a semi-automated screening tool to conduct the review efficiently at the title and abstract screening stage. In that case, the title and abstract of each citation will be reviewed by a team member, and then the AI Classifier Manager will serve as a second reviewer of each citation. The full text of each remaining potentially eligible article will be reviewed by a single team member to confirm eligibility and extract data. The team will decide whether it has enough time and resources to ask a second team member to check a randomly selected 10% sample of the articles to verify that important studies were not excluded and confirm the accuracy of extracted data.

Information will be organized according to the review questions, and will include author, year, study design, frequency and severity of the harms, measures of harm, characteristics of the PSP, rationale for the PSP, outcomes, implementation barriers and facilitators, required resources, and description of toolkits. To streamline data extraction, we will sort eligible studies by specific PSP (if the report covers more than one specific practice), and focus on extracting information about characteristics, outcomes, and barriers/facilitators most pertinent to a specific PSP.

For studies that address Review Question 5 about the effectiveness of PSPs, the primary reviewer will use the Cochrane Collaboration's tool for assessing the risk of bias of randomized controlled trials (RCTs) or the ROBINS-I tool for assessing the Risk Of Bias In Non-randomized—Studies of Interventions.^8,9 When assessing RCTs, we will use the 7 items in the Cochrane Collaboration's tool that cover the domains of selection bias, performance bias, detection bias, attrition bias, reporting bias, and other bias.⁸ When assessing non-randomized studies, we will use specific items in the ROBINS-I tool that assess bias due to confounding, bias in selection of participants into the study, bias in classification of interventions, bias due to deviations from intended interventions, bias due to missing data, bias in measurement of outcomes, and bias in selection of the reported results.⁹ The risk of bias assessments will focus on the main outcome of interest in each study.

If we identify a recent eligible systematic review, the primary reviewer will use the criteria developed by the United States Preventive Services Task Force Methods Workgroup for assessing the quality of systematic reviews.¹⁰

• Good—Recent relevant review with comprehensive sources and search strategies; explicit and relevant selection criteria; standard appraisal of included studies; and valid conclusions.
• Fair—Recent relevant review that is not clearly biased but lacks comprehensive sources and search strategies.
• Poor—Outdated, irrelevant, or biased review without systematic search for studies, explicit selection criteria, or standard appraisal of studies.

The Task Leader will review the risk of bias assessments and any disagreements will be resolved through discussion with the team.

Selected data will be compiled into evidence tables and synthesized narratively. We will not conduct a meta-analysis. For Review Question 5 about the effectiveness of PSPs, we will record information about the context of each study and whether the effectiveness of the PSP differs across patient subgroups. If any of the PSPs have more than one study of effectiveness, we will grade the strength of evidence for those PSPs using the methods outlined in the AHRQ Effective Health Care Program (EHC) Methods Guide for Effectiveness and Comparative Effectiveness Reviews.¹¹ Evidence grading would not add value for PSPs that do not have more than one available study.

If possible, for this rapid review, subgroup analyses will be conducted around effectiveness of CDS across different contexts, such as in-hospital or outpatient setting.

We will submit the protocol to AHRQ and to the PROSPERO international prospective register of systematic reviews.

EPC core team members must disclose any financial conflicts of interest greater than $1,000 and any other relevant business or professional conflicts of interest. Related financial conflicts of interest that cumulatively total greater than $1,000 will usually disqualify EPC core team investigators from participation in the review.

Peer reviewers are invited to provide written comments on the draft report based on their clinical, content, or methodological expertise. The EPC considers all peer review comments on the draft report in preparation of the final report. Peer reviewers do not participate in writing or editing of the final report or other products. The final report does not necessarily represent the views of individual reviewers.

We will ask at least one clinical content expert and one methodological expert to review the draft report. Potential peer reviewers must disclose any financial conflicts of interest greater than $5,000 and any other relevant business or professional conflicts of interest. Invited peer reviewers may not have any financial conflict of interest greater than $5,000.

This project is funded under Contract No. 75Q80120D00003/75Q80122F32009 from the Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services. The AHRQ Task Order Officer will review contract deliverables for adherence to contract requirements and quality. The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by AHRQ or the U.S. Department of Health and Human Services.

The report will follow the most recent template approved by AHRQ at the time of approval of the protocol.

1. Institute of Medicine Committee on Quality of Health Care in America. In: Kohn LT, Corrigan JM, Donaldson MS, eds. To Err is Human: Building a Safer Health System. Washington (DC): National Academies Press (US) Copyright 2000 by the National Academy of Sciences. All rights reserved; 2000.
2. Agency for Healthcare Research and Quality PSNet. Medication Errors and Adverse Drug Events. September 7, 2019. Accessed on May 8, 2023.
3. Bates DW, Levine DM, Salmasian H, et al. The Safety of Inpatient Health Care. N Engl J Med. 2023 Jan 12;388(2):142-53. doi: 10.1056/NEJMsa2206117. PMID: 36630622.
4. Gandhi TK, Weingart SN, Borus J, et al. Adverse drug events in ambulatory care. N Engl J Med. 2003 Apr 17;348(16):1556-64. doi: 10.1056/NEJMsa020703. PMID: 12700376.
5. Shojania KG, Duncan BW, McDonald KM, et al. Making health care safer: a critical analysis of patient safety practices. Evid Rep Technol Assess (Summ). 2001(43):i-x, 1-668. PMID: 11510252.
6. Shekelle PG, Wachter RM, Pronovost PJ, et al. Making health care safer II: an updated critical analysis of the evidence for patient safety practices. Evid Rep Technol Assess (Full Rep). 2013 Mar(211):1-945. PMID: 24423049.
7. Hall KK, Shoemaker-Hunt S, Hoffman L, et al. Making Healthcare Safer III: A Critical Analysis of Existing and Emerging Patient Safety Practices. Rockville (MD): Agency for Healthcare Research and Quality (US); 2020.
8. Higgins JP, Altman DG, Gotzsche PC, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011 Oct 18;343:d5928. doi: 10.1136/bmj.d5928. PMID: 22008217.
9. Sterne JA, Hernan MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016 Oct 12;355:i4919. doi: 10.1136/bmj.i4919. PMID: 27733354.
10. U.S. Preventive Services Task Force. U.S. Preventive Services Task Force Procedure Manual. Appendix VI. Criteria for Assessing Internal Validity of Individual Studies. U.S. Preventive Services Task Force. July 2017.
11. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ Publication No. 10(14)-EHC063-EF Agency for Healthcare Research and Quality. Rockville, MD: 2014.