Quality Control and Patient Risk Curtis A Parvin

Quality Control and Patient Risk Curtis A. Parvin, Ph. D. Manager of Advanced Statistical Research Quality Systems Division

Laboratory QC and Patient Risk • For the clinical laboratory patient risk is related to the reliability of the patient results they produce and report. • Laboratory Quality Control principles have been connected to the concept of patient risk for decades. • Specify the quality required of a patient result. • Determine the magnitude of “critical” out-of-control conditions with unacceptably high probability of producing patient results that fail to meet the required quality. • Design QC procedures with adequate power to detect “critical” out-of-control conditions. 2 |

Risk Management • A number of risk management guidelines have been published recently. • For manufacturers • ISO 14971: Medical devices – Application of risk management to medical devices • CLSI EP 18: Risk management techniques to identify and control laboratory error sources • For the laboratory • ISO 22367: Medical laboratories – Reduction of error through risk management and continual improvement • CLSI EP 23: Laboratory quality control based on risk management 3 |

The Risk Management Process • Risk Management provides a formal approach to • Identify potential failure modes in the lab • Rank the identified failure modes in terms of their risk • Establish policies and procedures to prevent or reduce (mitigate) the risks • Focus on the high ranked risks • This is generally a qualitative process • Risks are computed as a combination of • The probability of occurrence of harm • The severity of harm 4 |

Statistical QC and Risk Management • The question that we will address today is: How do statistical QC design principles that address questions such as • How many QCs should I examine? • What QC rule(s) should I use? • How frequently should I schedule QC evaluations? fit into the overall risk management process? 5 |

Risk Assessment (EP 23 Figure) Hazard Identification • • • Create a process map Identify potential failures in each process step Determine mechanisms in place to prevent or detect a failure Risk Estimation • • Assess the likelihood or probability of harm for each failure Assess the severity of harm to a patient from each failure Risk Control What control processes are needed to lowerrisk to an acceptable level No Risk Estimation Is the residual risk of harm clinically acceptable Yes The Laboratory’s QCP • • • 6 | Compile set of QC process into QCP Review QCP for conformance to regulatory and accreditation requirements Document and implement the set of control processes as the laboratory’s QCP

Risk Assessment (EP 23 Figure) Hazard Identification • • • Create a process map Identify potential failures in each process step Determine mechanisms in place to prevent or detect a failure Risk Estimation Assess the likelihood or probability of harm for each failure • Assess the severity of harm to a patient from each failure Risk Control What control processes are needed to lowerrisk to an acceptable level No Risk Estimation Is the residual risk of harm clinically acceptable Yes The Laboratory’s QCP • • • 7 | Compile set of QC process into QCP Review QCP for conformance to regulatory and accreditation requirements Document and implement the set of control processes as the laboratory’s QCP

Probability of Harm Sequence of Events Creating Risk of Harm for a Patient Hazardous Situation P 1 Initiating cause P 2 Testing process failure P 3 Incorrect result generated P 4 Incorrect result reported Misdiagnosis P 5 Hazardous medical action P 6 Patient harmed EP 23 -A, Figure 6 ISO 14971, Figure H. 1 ISO/TS 22367, Figure B. 1 • Estimates of the probability of harm are desirable • For each potential mode of failure • Collectively for all possible failure modes 8 |

Estimating Probability of Harm P 1 Initiating cause P 2 Testing process failure P 3 Incorrect result generated P 4 Incorrect result reported Misdiagnosis P 1: The probability that a testing process failure occurs 9 | P 5 P 6 Hazardous medical action Patient harmed

P 1: Probability of a Failure Mode • P 1 = how likely is a given failure mode going to occur or more commonly • P 1 can be the mean time between failures (MTBF) Information regarding MTBF may come from • Manufacturer • Historical failure data • Literature • MTBF can be converted to mean number of patient results between failures (MPBF) • MTBF = 30 days • 150 patient results examined per day (on average) • MPBF = 30 * 150 = 4500 patient results 10 |

Estimating Probability of Harm MPBF P 1 Initiating cause Testing process failure P 2 Incorrect result generated P 3 P 4 Incorrect result reported P 5 Misdiagnosis P 6 Hazardous medical action Patient harmed P 2: The probability that a testing process failure generates incorrect patient results 11 |

P 2: Probability of Producing an Incorrect (Unreliable) Result • The number of unreliable patient results produced during the existence of an out-of-control condition depends on • The quality required for patient results • The type and magnitude of the out-of-control condition • The power to detect the out-of-control condition • When (how often) QC evaluations are performed • An out-of-control condition may result in many unreliable patient results being produced 12 |

Unreliable Patient Results • ISO 15189 Clause 5. 6. 1: Laboratory QC should assure that patient results meet the quality required for their intended use • The quality of a patient result depends on the difference between the correct value and the value reported. • If the error in a patient’s result exceeds the allowable total error (TEa) • The result is considered unreliable (or incorrect). • It creates a hazardous situation for the patient. 13 |

Expected Number of Unreliable Patient Results Produced: E(NU) E(Nu) • The number of unreliable patient results produced during an out-of-control condition (red asterisks) will depend on • The magnitude of the out-of-control condition • The power of the QC rule – Detection • The number of patient specimens between QC events • E(Nu) = Expected number of unreliable patient results generated during an out-of-control condition 14 |

Estimating Probability of Harm MPBF Initiating cause Testing process failure E(N P 2 u)u) Incorrect result generated P 3 P 4 Incorrect result reported P 5 Misdiagnosis P 6 Hazardous medical action P 3: The probability that incorrect results that are generated will be reported 15 | Patient harmed

P 3: Probability of Reporting an Unreliable Patient Result • The probability of reporting an unreliable result that leads to an incorrect action depends on • The number of unreliable results produced because of an out-of-control condition • How and when results are reported • The likelihood of identifying and correcting a reported unreliable result before an incorrect action is taken 16 |

Expected Number of Unreliable Patient Results Reported E(Nu) E(Nuf) E(Nuc) • The patient results produced during an out-of-control condition can be divided into • Results prior to the last accepted QC event (Pre) • Results since the last accepted QC event (Post) • Bracketed QC: results aren’t reported • Immediate reporting: results should be repeated and updated in a timely fashion • E(Nuf) = Expected number of unreliable “final” results • E(Nuc) = Expected number of unreliable “correctable” results 17 |

Estimating Probability of Harm MPBF Initiating cause Testing process failure E(Nu) E(N P 3 ufuf) ) Incorrect result generated Incorrect result reported P 4 P 5 Misdiagnosis Hazardous medical action P 6 Patient harmed P 4: The probability that an incorrect result leads to a misdiagnosis P 5: The probability that a misdiagnosis leads to a hazardous medical action P 6: The probability that a hazardous medical action leads to patient harm P 4*P 5*P 6: The probability than an incorrect result leads to patient harm 18 |

Probability of Patient Harm from an Unreliable Patient Result • The probabilities (P 4, P 5, P 6) are associated with activities outside the laboratory • Their product equals the probability of patient harm given that an unreliable result has been reported Ph|u = P 4*P 5*P 6 • Estimates of the probability of harm given an incorrect result are based on; • Medical literature • Medical judgment • Consultation with clinical colleagues 19 |

Estimating Probability of Harm MPBF Initiating cause 20 | Testing process failure E(Nu) P 4 E(Nuf) Incorrect result generated Incorrect result reported P 5 P h|u Misdiagnosis Hazardous medical action P 6 Patient harmed

Estimating Probability of Harm: Risk Management + Statistical QC Risk Management Statistical QC MPBF Initiating cause 21 | Testing process failure E(Nu) E(Nuf) Incorrect result generated Ph|u Incorrect result reported Patient harmed

Harmed Patients from Failure Mode Occurrence Ph|u*E(Nuf) MPBF * * E(Nuf) * E(Nuc) E(Nu) • Probability of patient harm is • The expected number of patients harmed due to incorrect results from a failure mode: Ph|u* E(Nuf) • Divided by the average number of patient results examined between failure mode occurrences: MPBF Probability of Harm = Ph|u* E(Nuf) / MPBF 22 |

What Info Do We Need to Predict E(Nu) and E(Nuf)? • The quality requirement for the analyte Needed to compute the • Allowable Total Error (TEa) probability of an • Test method performance unreliable patient result Needed to compute the • Instrument imprecision and bias expected # of • QC procedure affected patient results • QC rule(s) used • Number of QC levels evaluated at each QC event • QC schedule (average number of patients examined for the analyte between QC events) 23 |

Probability of an Unreliable Result Due to a Systematic Error Condition 24 |

Probability of an Unreliable Result Depends on TEa, Method CV and Bias 25 |

Expected # Affected Depends on QC Rule and QC Schedule 26 |

Expected # of Unreliable Patient Results 27 |

Ways to Reduce (or Manage) Patient Risk Estimate Risk Reduce Risk Rate of Occurrence of Failure Modes Yes: manufacturer info, technical bulletins, product alerts Some: lab environment/processes training Very little: Medical devices Number of Incorrect Results Reported Yes: Statistical QC design & evaluation A lot: Detection in analytical phase A little: Detection in pre- and postanalytical phases Probability of Yes: Medical literature, Harm from medical judgment, an Incorrect consultation Result 28 | Very little: Report formatting, education

Ways to Reduce (or Manage) Patient Risk • Manage the risk of unreliable patient results in the worst case • Limit the maximum values for E(Nuf) and E(Nuc) • Manage the overall expected risk of unreliable patient results • Limit the areas under the E(Nuf) and E(Nuc) curves • Manage the “defect rate” for unreliable patient results • Requires an estimate of the rate at which out-of-control conditions occur 29 |

Summary • The laboratory has two main mechanisms to estimate and reduce the risk of patient harm: 1. Identify as many potential failure modes as possible and seek ways to reduce occurrences of the identified failure modes Risk management 2. Implement QC strategies that minimize the number of incorrect patient results that are reported when a failure mode does occur Statistical QC • The combination enables a laboratory to address the full spectrum of patient risk implications for their operations 30 |

References • • 31 | Parvin CA. Assessing the impact of the frequency of quality control testing on the quality of reported patient results. Clin Chem 2008; 54: 2049 -54. Parvin CA, Yundt-Pacheco J, Williams M. Analytical assessment in the clinical laboratory: assessing analytical quality goals when the same analyte can be tested on multiple systems is explored. Adv Admin Lab 2011; 20(1): 28 -31. Parvin CA, Yundt-Pacheco J, Williams M. The focus of laboratory quality control: why QC strategies should be designed around the patient, not the instrument. Adv Admin Lab 2011; 20(3): 48 -9. Parvin CA, Yundt-Pacheco J, Williams M. Designing a quality control strategy: in the modern laboratory there are 3 questions that must be answered. Adv Admin Lab 2011; 20(5): 53 -4. Parvin CA, Yundt-Pacheco J, Williams M. The frequency of quality control testing: QC testing by time or number of patient specimens and the implications for patient risk are explored. Adv Admin Lab 2011; 20(7): 66 -9. Parvin CA, Yundt-Pacheco J, Williams M. Recovering from an out-of-control condition: the laboratory must assess the impact and have a corrective action strategy. Adv Admin Lab 2011; 20(11): 42 -4. Parvin CA, Yundt-Pacheco J, Williams M. Sigma metrics, total error budgets, and quality control: Make sure your test system performance and quality control procedures are aligned with your quality goals. Adv Admin Lab 2012; 21(1): 40 -4.

32 |

33 |

THANK YOU! 34 |