Challenges using Safety Monitoring Systems A review of

Challenges using Safety Monitoring Systems A review of Integrating Incident Data from Five Reporting Systems to Assess Patient Safety: Making Sense of the Elephant David Peavy Carter Seward Jake Swinton

Reference Osnat Levtzion-Korach, M. D. M. H. A. ; Allan Frankel, M. D. ; Hanna Alcalai, B. P. T. , M. H. A; Carol Keohane B. S. N. , R. N. ; John Orav, Ph. D. ; Erin Graydon-Baker, M. S. , R. R. T. ; Janet Barnes, R. N, J. D. , C. P. H. R. M. ; Kathleen Gordon, M. S. N. , M. S. ; Anne Louise Puopulo, B. S. N, R. N. ; Elena Ivanova Tomov, M. B. A. ; Luke Sato, M. D. ; David W. Bates, M. D. , M. Sc. (September 2010). Integrating Incident Data from Five Reporting Systems to Assess Patient Safety: Making Sense of the Elephant. The Joint Commission Journal on Quality and Patient Safety, Vol. 36, No. 9. Retrieved from: http: //psnet. ahrq. gov/public/JCJQPS 2010 -PSNet-ID-19031. pdf. Accessed February 6, 2015. Safety Monitoring Systems ETM 568 - Peavy, Seward & Swinton 2

Introduction ● The article investigates the various reporting systems and how they correlate to one another o o Use different methods of obtaining information Place results using different categories Different personnel fill our reports Prioritize different events ● The article reviewed the quality reporting systems/processes used at Brigham and Women’s Hospital (BWH) located in Boston, MA. o o o 747 -bed tertiary care academic medical center 52, 000 inpatient admissions/950, 000 outpatient admissions per year 12, 000 employees; approximately 3000 are doctors Safety Monitoring Systems ETM 568 - Peavy, Seward & Swinton 3

Methods ● BWH uses a total of 5 Quality Reporting Systems: o o o ● Incident Reporting Reports to Risk Management Patient Complaints Executive Walk Arounds Malpractice Claims The authors wanted to: o o Evaluate what type of information is received by each system Develop a common framework for representing the identified safety issues Assess the correlation between types of information collected Evaluate the overall safety picture Safety Monitoring Systems ETM 568 - Peavy, Seward & Swinton 4

Data ● The data was collected from the source systems over a 22 month period between May 10 th, 2004 and February 28 th, 2006 ● The 22 month time period had a small number of malpractice claims, so to increase this number, the time period was extended to 10 years. Safety Monitoring Systems ETM 568 - Peavy, Seward & Swinton 5

Assessment Areas ● Incident Reporting System o o o Commercially available web- based incident reporting system Usually filled out by nurses Evaluated by Department Managers ● Hospital Risk Management o o o Physicians and Nurses call the team to report adverse events and poor patient outcomes Even split of reporting between Nurses and Physicians Evaluated by Risk Management staff Safety Monitoring Systems ETM 568 - Peavy, Seward & Swinton 6

Assessment Areas (Cont. ) ● Patient Complaints o ● Executive Walk Around o ● The Family and Patient Relations Department Responds to patient and family complaints, suggestions, and compliments Executive leadership walk arounds occur semiweekly and engage with staff about safety concerns Malpractice Claims o o The malpractice insurer has a system used for data collection called CMAPS. This system takes information from potential claim reports risk managers, and formal malpractice claims Safety Monitoring Systems ETM 568 - Peavy, Seward & Swinton 7

Results (Page 404; Levtzion-Korach, M. D. et. al. ) Safety Monitoring Systems ETM 568 - Peavy, Seward & Swinton 8

Comparisons ● The five assessment areas did not have overlapping categorization schemes ● The experimenters developed a categorization scheme with 23 major categories to compare the 5 reporting systems. Safety Monitoring Systems ● Examples: o Staff Training/Education o Technical Skills o Equipment/Supplies o Medication Errors o Identification Issues o Monitoring of Patient o Clinical Judgement o Medical Records o Test Reconciliation Issues ETM 568 - Peavy, Seward & Swinton 9

Analysis ● Data collected by the five reporting systems was analyzed and classified into one of the new categories by a physician reviewer trained in patient safety. ● Investigators calculated the frequencies of each of the events ● Overlap between rankings of frequency of different categories were then assessed and correlations between the detection approaches were calculated Safety Monitoring Systems ETM 568 - Peavy, Seward & Swinton 10

Categorical Analysis ● The leading major categories across the reporting systems were: o o o Communication (11. 6%) Technical Skills (10. 9% Clinical Judgement (9%) ● However, each system had a different category that was most frequent Safety Monitoring Systems ETM 568 - Peavy, Seward & Swinton 11

Statistical Correlation (Page 408; Levtzion-Korach, M. D. et. al. ) ● Overall, across the five systems, Cronbach’s standardized alpha was 0. 22, suggesting a low level of consistency Safety Monitoring Systems ETM 568 - Peavy, Seward & Swinton 12

Correlations ● Correlations presented: o Highest correlation observed was between risk management reports and malpractice claims o A high correlation was found between patient complaints and malpractice claims o The incident reporting system has little correlation with the other systems o Executive walk rounds had a negative correlation with all other systems Safety Monitoring Systems ETM 568 - Peavy, Seward & Swinton 13

(Page 403; Levtzion-Korach, M. D. et. al. ) Safety Monitoring Systems ETM 568 - Peavy, Seward & Swinton 14

(Page 403; Levtzion-Korach, M. D. et. al. ) Safety Monitoring Systems ETM 568 - Peavy, Seward & Swinton 15

Discussion ● Some things to keep in mind about the study: o o Only a single medical center was used. Results shown may not be generalizable to other facilities Not all possible types of safety data were included. However, BWH allowed the authors to closely examine numerous independent data source Safety Monitoring Systems ETM 568 - Peavy, Seward & Swinton 16

Discussion (cont’d) ● The study allowed for the comparison and evaluation of each reporting system, both individually, and as a whole ● Each system provides a different set of partial conclusions, and there is little correlation between individual systems Safety Monitoring Systems ETM 568 - Peavy, Seward & Swinton 17

Conclusion ● Studies or reports commenting on only one of the reporting systems are incomplete ● Much better, more complete conclusions may be drawn from a complete perspective that utilizes information from multiple reporting systems ● Hospitals should utilize a more broad approach to the classification of adverse events in order to make more informed decisions Safety Monitoring Systems ETM 568 - Peavy, Seward & Swinton 18