Usability and Human Factors Unit 7 Decision Support
Usability and Human Factors Unit 7: Decision Support Systems: a Human Factors Approach Lecture c This material (Comp 15 Unit 7) was developed by Columbia University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number 1 U 24 OC 000003. This material was updated by The University of Texas Health Science Center at Houston under Award Number 90 WT 0006. This work is licensed under the Creative Commons Attribution-Non. Commercial-Share. Alike 4. 0 International License. To view a copy of this license, visit http: //creativecommons. org/licenses/by-nc-sa/4. 0/.
Decision Support Systems: a Human Factors Approach Lecture c – Learning Objectives • Describe the cognitive basis for decision making and its effect on clinical errors (Lecture a) • Describe the role and advantages of Clinical Decision Support Systems (CDSS) (Lecture b) • Discuss the role of usability testing, training and implementation of clinical decision support (Lecture c) • Describe and define usability as it pertains to clinical decision support (Lecture c) • Identify examples of usability barriers to adoption of clinical decision support (Lecture d) 2
A Drug-Drug Interaction Scenario • “When ordering a new medication, a prescriber may not be aware that two drugs interact, or may not be keeping in mind the other medications that the patient is taking. • As an example, consider the case of a hospitalized patient who is being treated with venlafaxine (Effexor) for chronic depression and develops an infection with a drug resistant bacterium requiring treatment with linezolid, a new antimicrobial agent. • The interaction between linezolid and venlafaxine (serotonin syndrome -- altered mental status, including agitation, confusion and coma, neuromuscular hyperactivity, and autonomic dysfunction) is very severe but may not be known to the practitioner. 3 Kuperman, G. , et al, (2007)
A Drug-Drug Interaction Scenario (Cont’d – 1) • While writing the order for linezolid, an alert screen can warn the practitioner that these two drugs should not be used together. • The alert screen may offer the prescriber the opportunity to cancel the order, to discontinue the existing medication that interacts with the newly ordered medication, or to order a test that could detect the interaction or monitor therapy. The alert screen may prompt the physician to have a conversation with the patient regarding potential side effects of the medications. Any of these consequences of the decision support software could be beneficial. ” Kuperman , G. , et al, (2007) 4
Challenges with Order Entry • Steep learning curve • Disruption of workflow and organization roles and routines • Perceived increase in completion time – Decreases with gains in expertise • Can introduce new sources of error 5
CPOE Paradox • CPOE identified as important intervention to reduce prescribing errors and yet the evidencebase for their effectiveness is limited • Some studies have shown electronic prescribing with CPOE significantly increases prescribing quality in hospital inpatients • Also introduces new types of errors which have been introduced following CPOE system implementation 6
Cognitive Evaluation of Interaction with a CDSS • Cognitive evaluation of an interaction with a CPOE system – Focus on decision support for heparin dosing • Objective: – Characterize the interaction in terms of effectiveness – Changes to ordering behavior – Opportunities for error attributable to the interaction process • Representational format of information in alerts affects performance and errors – Timing of trigger in relation to workflow (Horsky, . et al, 2005) 7
Weight-based Herparin Ordering Clinical scenario for the administration of weight-based heparin • Decide on giving bolus followed by IV drip • Calculate both doses CPOE triggers decision-support alert • Calculates dose automatically 8
Methods Cognitive walkthrough Entry of hospital admission orders • Step-by-step task analysis – goals, actions • Completed by 2 researchers • Characterize information to complete task • Completed by 7 clinicians • Subjects instructed to think aloud • Screen progression captured on video • Transcripts of comments coded for analysis 9
CPOE Screen Horsky, J. , Kaufman, D. R. , & Patel, V. L. (2005) 10
Decision Support Alert Weightbased IV Heparin Protocols Horsky, J. , Kaufman, D. R. , & Patel, V. L. (2005). 11
Results – Presentation Salience Recognition of DS purpose Visual salience of calculated values • 2 clinicians confused • Values cleared from patient-specific dose screen when users calculator with a needed to enter them general guideline – memory recall • 6 clinician engaged in extended reading of text to derive alert meaning 12
Results – User Behavior • Knowledge of DS availability – 1 subject computed both doses before DS was triggered – no indication of DS function • Dose estimated before DS – 6 subjects used system-calculated value only as reference – extra time, cognitive effort • Transparency of computation – 3 subjects guessed system is using “ 80/18” formula to compute weight-based dose 13
Summary • Suboptimal presentation format – Extra time, few realized benefits of DS – Inconsistent with workflow • Different representational form would enable a quick perceptual judgment could reduce this extra cognitive effort – Leaving only the calculated dose in the frame with a clear description of how the result was computed – To take advantage, user needs to be able to invoke this feature on demand – May identify inefficient and error-prone screen configurations • Propose specific interface improvement – Higher rate of decision support use 14
Study: Role of CPOE Systems in Facilitating Medical Errors • Koppel et al (2005) conducted multifaceted study investigating medication errors associated with CPOE use • CPOE system facilitated 22 types of medication error. – Many appeared to occur with great frequency 15
Role of CPOE Systems Study: Medical Error Classification • Errors classified into: – Information errors generated by fragmentation of data and failure to integrate the hospital’s information systems – human-machine interface flaws reflecting machine rules that do not correspond to work organization or usual behaviors 16
Information Errors: Fragmentation and Systems Integration Failure • Assumed Dose Information: – House staff rely on CPOE displays to determine minimal effective or usual doses. – dosages listed in display based on the pharmacy’s warehousing and not clinical guidelines. – For example, normal dosages are 20 or 30 mg, the pharmacy might stock only 10 -mg doses, so 10 -mg units are displayed on the CPOE screen – Clinicians select inappropriate doses 17
Information Errors: Fragmentation and Systems Integration Failure (Cont’d – 1) • Medical Discontinuation Failures: – Ordering new or modifying existing medications is usually a separate process from canceling – Without discontinuing the current dose, physicians can increase or decrease medication – Add new but duplicative medication – Medication-canceling ambiguities exacerbated by interface and multiple-screen displays of medications o Viewing 1 patient’s medications may require 20 screens 18
Human-Machine Interface Flaws • 1. Patient Selection – Easy to select wrong patient file because names and drugs are close together, the font is small, and, patients’ names do not appear on all screens • 2. Unclear Log On/Log Off – Physicians can order medications at computer terminals not yet "logged out" by the previous physician – result in either 1) unintended patients receiving medication or 2) patients not receiving the intended medication 19
Human-Machine Interface Flaws (Cont’d – 1) 3. Failure to Provide Medications After Surgery – When patients undergo surgery, CPOE cancels their previous medications. – Physicians must reenter CPOE and reactivate each previously ordered medication 20
Automation Bias • Clinicians using automated decision follows the aid directs regardless of knowledge to the contrary • Errors of Omission: – Less vigilant in checking drug orders because they assume the computer will have already done the work • Errors of Commission – Continue with a dangerous drug order because the computer did not alert them that the order was unsafe 21
Anti-Automation Bias • Errors of dismissal, where computer advice is ignored • Clinicians routinely disable or ignore the alarms or alerts on clinical monitoring devices – Legitimate reasons such as high false alarm rates [or repetition of the same alarms] – Less valid reasons such as not wanting to be interrupted 22
Unit 7: Decision Support Systems: a Human Factors Approach Summary – Lecture c • CDSS have also been shown to introduce new types of errors in the decision making process • Usability testing and cognitive walkthrough analysis have been conducted by expert analysts to study CDSS • In the study discussed, CPOE system facilitated 22 types of medication error 23
Unit 7: Decision Support Systems: a Human Factors Approach References – Lecture c References: Ash, J. S. , Sittig, D. F. , Campbell, E. M. , Guappone, K. P. , & Dykstra, R. H. (2007, October). Some unintended consequences of clinical decision support systems. In AMIA 2007 Symposium Proceedings, pp. 26 -30 Kuperman, G. J. , Bobb, A. , Payne, T. H. , Avery, A. J. , Gandhi, T. K. , Burns, G. , et al. (2007). Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc, 14(1), 29 -40. Coiera E, Westbrook J, Wyatt J. The safety and quality of decision support systems. Methods Inf Med. 2006; 45 Suppl 1(suppl 1): 20– 5. me 06010020 Eddy, D. M. (1982). Probabilistic reasoning in clinical medicine: Problems and opportunities. In D. Kahneman, P. Slovic & A. Tversky (Eds. ), Judgment under uncertainty: Heuristics and biases (pp. 249 -267). Cambridge, England: Cambridge University Press. Health. IT. gov. (n. d. ). Retrieved April 26, 2016, from https: //www. healthit. gov/policy-researchers-implementers/clinicaldecision-support-cds Horsky, J. , Kaufman, D. R. , & Patel, V. L. (2005). When you come to a fork in the road, take it: strategy selection in order entry. AMIA Annu Symp Proc, 350 -354. Horsky, J. , Schiff, G. D. , Johnston, D. , Mercincavage, L. , Bell, D. , & Middleton, B. (2012). Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions. Journal of biomedical informatics, 45(6), 1202 -1216. Institute of Medicine Crossing the Quality Chasm: A New Health System for the 21 st Century. Washington, DC: National Academy Press, 2001. J. A. Osheroff, J. M. Teich, B. Middleton, E. B. Steen, A. Wright and D. E. Detmer, A roadmap for national action on clinical decision support, J Am Med Inform Assoc 14 (2) (2007), pp. 141– 145. Karsh B-T. Clinical practice improvement and redesign: how change in workflow can be supported by clinical decision support. AHRQ Publication No. 09 -0054 EF; Rockville (MD): Agency for Healthcare Research and Quality; June 2009. 24
Unit 7: Decision Support Systems: a Human Factors Approach References – Lecture c (Cont’d – 1) References Khajouei R, Jaspers MW: CPOE system design aspects and their qualitative effect on usability. Stud Health Technol Inform 2008 , 136: 309 -14 Koppel, R. , Metlay, J. P. , Cohen, A. , Abaluck, B. , Localio, A. R. , Kimmel, S. E. , et al. (2005). Role of computerized physician order entry systems in facilitating medication errors. JAMA, 293(10), 1197 -1203. Kuperman, G. J. , Bobb, A. , Payne, T. H. , Avery, A. J. , Gandhi, T. K. , Burns, G. , et al. (2007). Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc, 14(1), 29 -40. Mc. Neil, B. J. , Pauker, S. G. , Sox, H. C. , Jr. , & Tversky, A. (1982). On the elicitation of preferences for alternative therapies. N Engl J Med, 306(21), 1259 -1262. Reckmann, M. H. , Westbrook, J. I. , Koh, Y. , Lo, C. , & Day, R. O. (2009). Does computerized provider order entry reduce prescribing errors for hospital inpatients? A systematic review. J Am Med Inform Assoc, 16(5), 613 -623. Russ, A. L. , Zillich, A. J. , Mc. Manus, M. S. , Doebbeling, B. N. , & Saleem, J. J. (2009). A human factors investigation of medication alerts: barriers to prescriber decision-making and clinical workflow. AMIA Annu Symp Proc, 2009, 548552. Shortliffe, E. H. , & Cimino, J. J. (2013). Biomedical informatics: computer applications in health care and biomedicine. Springer Science & Business Media. 4 th edition. Teich JM, Osheroff JA, Pifer EA, Sittig DF, Jenders RA, Panel CDSER. Clinical decision support in electronic prescribing: Recommendations and an action plan. J Am Med Inform Assoc 2005; 12(4): 365 -76. Images Slide 10: Horsky, J. , Kaufman, D. R. , & Patel, V. L. (2005). When you come to a fork in the road, take it: strategy selection in order entry. AMIA Annu Symp Proc, 350 -354 Slide 11: Horsky, J. , Kaufman, D. R. , & Patel, V. L. (2005). When you come to a fork in the road, take it: strategy selection in order entry. AMIA Annu Symp Proc, 350 -354 25
Unit 7: Decision Support Systems: a Human Factors Approach Lecture c This material was developed by Columbia University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number 1 U 24 OC 000003. This material was updated by The University of Texas Health Science Center at Houston under Award Number 90 WT 0006. 26
- Slides: 26