ASSESSMENT AUDITS Each year more than 1 million

ASSESSMENT— AUDITS

Each year, more than 1 million preventable injuries and 44, 000– 98, 000 preventable deaths occur in the United States alone. In the United States, approximately 2 trillion dollars are spent on medical care each year, and the health standards are higher than in many other countries. Therefore, it is assumed that preventable injury and death rates due to medical errors in many countries are higher than those in the United States 2

Definitions Quality system: The defined organizational structure, responsibilities, processes, procedures and resources for implementing and coordinating the quality assurance and quality system audit. Quality system essentials: The necessary infrastructure in any organization that need to be in place and functioning effectively in order to support the organization’s work operations so that they proceed smoothly. Quality management: Coordinated activities that managers carry out in an effort to implement their quality policy. These activities include quality planning, quality control, quality assurance and quality improvement. Quality management system: Management system to direct and control an organization with regard to quality. 3

Introduction Assessment is an important element of the quality system essentials. It is the means for determining the effectiveness of a laboratory’s quality management system through internal and external audits, and evaluation of performance in an external quality assessment (EQA) program. This chapter is focused on descriptions of internal and external audits. 4

What is Assessment? An assessment can be defined as the systematic examination of some part (or sometimes all) of the quality management system to demonstrate to all concerned that the laboratory is meeting regulatory, accreditation and customer requirements. Accepted standards, whether international, local, or standards from accrediting organizations, form the basis for laboratory assessment. 5

What is Assessment? In that respect, assessment is interrelated with standards and accreditation. In an assessment, someone is asking the following questions: What procedures and processes are being followed in the laboratory; what is being done? Do the current procedures and processes comply with written policies and procedures? And in fact, are there written policies and procedures? Do written policies and procedures comply with standards, regulations, and requirements? 6

Why Perform an Assessment? An assessment, or audit, allows the laboratory to understand how well it is performing when compared to a standard. Any gaps or nonconformities in performance can show if the policies and procedures that the laboratory has set require revision or are not being followed. A laboratory needs this information about its performance for Planning and implementing the quality system Monitoring effectiveness of the quality system Correcting any deficiencies that are identified Working toward continuous improvement. 7

External & Internal Audits Assessments conducted by groups or agencies from outside the laboratories are called external audits. They can include assessments for the purpose of accreditation or licensure. Another type of assessment that laboratories can utilize is the internal audit, where staff working in one area of the laboratory conduct assessments on another area of the same laboratory. This provides information quickly and easily on how the laboratory is performing and whether it is in compliance with policy requirements. 8

Auditing The value of a well-designed audit is that it will reveal weaknesses in the preanalytical, analytical and postanalytical phases. During audits, information is gathered about: Processes and operating procedures Staff competence and training Equipment; Environment; handling of samples Quality control and verification of results Recording and reporting practices. The findings are compared with the laboratory’s internal policies and to a standard or external standard. Any breakdown in the system or departure from procedures will be identified. 9

External Audit Assessments conducted by groups or agencies from outside the laboratory are called external audits. Some examples of external auditors are: Health authorities may assess laboratories to evaluate the quality of performance, or compliance with licensing requirements and national regulations. Accreditation bodies are organizations that provide accreditation or certification. An audit may be requested agencies that provide funding for programs. 10

External Audit In conducting external audits, the assessors will verify that laboratory policies and procedures are documented and comply with designated standards. After the audit, the recommendations of the assessors are often presented as a verbal summary to the laboratory management and staff, which are then followed by a thorough written report. 11

External Audit After the external audit has been completed the laboratory should: Review the recommendations of the assessors; Identify gaps or nonconformities, learning where standards were not fully met; Plan to correct the nonconformities—this will result in a plan for all needed corrective actions to be taken by the laboratory, which should include a timeline, as well as indicate who is responsible for doing the work; Record all results and actions taken so that the laboratory has a permanent record of the event—often a written report is useful for preserving all information. 12

Standardization Bodies ISO (International Organization for Standardization) The world's largest developer and publisher of international standards, and ISO standards are applicable to many kinds of organizations, including clinical and public health laboratories. ISO 15189 is a preferred standard for medical laboratories because it applies to the total laboratory, regardless which tests it performs CLSI (Clinical and Laboratory Standards Institute) CLSI is a global, non-profit, standards-developing organization that promotes the development and use of voluntary consensus standards and guidelines within the health care community. 13

Standardization Bodies CEN (European Committee for Standardization) CEN was founded in 1961 by the national standards bodies in the European Economic Community and associated countries. CAP (The College of American Pathologists) Inspects a variety of laboratory settings from complex university medical centers to physician office laboratories, and covers a complete array of disciplines and testing procedures. Joint Commission Works to improve patient safety and quality of health care in the international community by offering education, publications, advisory services, and international accreditation and certification. 14

15

Internal Audit An internal audit allows the laboratory to look at its own processes. In contrast to external audits, the advantages of internal audits are that laboratories can perform them as frequently as needed, and at very little or no cost. Internal audits should be a part of every laboratory quality system, and are a requirement of ISO standards. The audits should be conducted regularly and when problems occur. 16

Internal Audit For example, internal audits should be performed: After receiving a poor performance on a proficiency testing survey, After an increased number of unexpected abnormal results for a particular test, or After an increase in expected turnaround time. 17

Value of an Internal Audit An internal audit can help the laboratory to: Prepare for an external audit; Increase staff awareness of quality system requirements; Identify the gaps or nonconformities that need to be corrected—the opportunities for improvement; Understand where preventive or corrective action is needed; Identify areas where education or training needs to occur; Determine if the laboratory is meeting its own quality standards. 18

Establishing a Process for Internal Audits The quality manager or other designated qualified personnel should organize the internal audit following these steps: Develop a formal plan Prepare a checklist based on selected guidelines or standards Meet with all staff and explain the audit process Select staff to serve as auditors Collect and analyze information Share results with staff Prepare a report Present the report to management Retain the report as a permanent laboratory record 19

Select Areas for Audits In order to facilitate the internal audit process, it is useful to keep it simple. Focus on defined areas of the laboratory activities, identified by issues such as customer complaints or quality control problems. Narrowing the audit to the specific corresponding process will save time and energy. Perform short and frequent audits rather than initiating an annual comprehensive and massive effort. 20

Actions as a Result of Audits should lead to actions—this is why laboratories conduct them, to further the process of continual improvement in the laboratory. Audits identify opportunities for improvement (OFIs). Both preventive and corrective actions are steps taken to improve a process or to correct a problem. A record of OFIs should be kept, along with actions that are taken. Preventive and corrective actions should be carried out within an agreed-upon time. Normally the quality manager is responsible for initiating actions. 21

Continuous Monitoring Continuous monitoring is the key element to success in the quality system. It is through this process that we are able to achieve the continual improvement that is our overall goal. 22

QUALITY IMPROVEMENT

Introduction The continual improvement of the laboratory processes is essential in a quality management system. There is a need to be constantly working toward making the process better. Perfection is never achieved, but we try to get as close to it as possible. Process improvement is something that is never finished, but rather continues on “forever”. Quality improvement goes beyond monitoring, detecting, and preventing errors. Quality improvement achieves new levels of performance, not otherwise realized through QC and addresses chronic problems. 24

What is Process Improvement? A process is a series of actions or operations contributing to an end. In every case, inputs (patient samples) are turned into outputs (patient examination results) because some kind of work, activity or function is carried out. Process improvement is a systematic and periodic approach to improving laboratory quality and the inputs, the outputs and the activities. It is a way of solving problems. 25

Conventional Tools for Improvement Many useful techniques have been developed to use in process improvement. For example: Both internal and external audits will identify system weaknesses and problem areas. Participation in an external quality assessment is another useful tool; it allows for comparing laboratory performance to that of other laboratories. There should be management reviews of the laboratory records on a regular basis; for example, quality control, inventory management and equipment maintenance. 26

27

Quality Indicator What is a quality indicator? Established measures used to determine how well an organization meets needs and operational and performance expectations. Purpose of quality indicators: Give information about the performance of a process Determine quality of services Highlight potential quality concerns Identify areas that need further study and investigation Track changes over time. 28

Tools for Continual Improvement New ideas for tools to use for continual improvement continue to come from the manufacturing industry. Two of these new tools are now being used in laboratory quality improvement: Six Sigma Lean 29

Six Sigma methodology represents an evolution in quality assessment and management that has been implemented widely in business and industry since the mid-1980 s. Six Sigma methodology was developed by Motorola, to reduce the cost of products, eliminate defects, and decrease variability in processing. Six Sigma is a quality management strategy that seeks to improve assay quality by identifying biased and/or imprecise assays so that appropriate quality monitoring strategies can be used and assay performance can be improved. In Six Sigma, the focus is to move toward reducing error to very low levels. 30

Adopting Six Sigma as the Goal for Laboratory Testing Six Sigma uses a problem-cause-solution methodology to improve any process through waste elimination and variation reduction. The DMAIC (Define, Measure, Analyze, Improve, and Control) methodology is more than a series of phases and steps, or tools and techniques. It is the belief that quality improvement requires right problem solving. The DMAIC methodology is a quality improvement team’s project management road map. 31

DMAIC Methodology 32

DMAIC Methodology (D) Define. This process involves determining: Who uses your services or products? For example, this might be mainly the physicians within a hospital. What are the user’s requirements and expectations? For example, an emergency department will have a need for a faster turnaround time to treat acutely ill patients than will a family physician’s office. What are the project boundaries? This might involve establishing who is responsible for the different parts required for generated test results (e. g. , phlebotomy to draw the specimen). The process to be improved by methods such as mapping the process flow. This involves determining how a process currently works by mapping all of the steps involved in performing the task. 33

DMAIC Methodology (M) Measure: The performance of a process in the laboratory can be measured by: Collecting data. In the laboratory, this might involve determining how long it takes to report the results of a cardiac marker from a specimen collected in the emergency department. Determine defects. For example, this might include determining how often specimens are collected in the wrong tube. Satisfaction determined by surveys of those using the process. This might involve determining if the needs of the physicians (or other end-users of test results) are being met. 34

DMAIC Methodology (A) Analyze: Examine the data collected including the process map to identify the root causes of errors in the process being investigated. (I) Improve: Improve the process by creative solutions to fix problems and prevent future ones from occurring. (C) Control: This step keeps the implemented improvements from being forgotten or ignored by long term monitoring of the process over time. This may include continuous monitoring of the new plans. 35

Adopting Six Sigma as the Goal for Laboratory Testing The goal of Six Sigma is to eliminate or reduce all variation in a process. For example, variation in a process leads to wasted effort and resources on retesting and workarounds. Reducing defects reduces costs, and improves performance and profitability. A process that achieves the goal of Six Sigma delivers both quality and efficiency. 36

Sigma Metrics In practice, a process is evaluated in terms of a sigma metric that describes how many sigmas fit within the tolerance limits. Six Sigma is a metric that quantifies the performance of processes as a rate of Defects-Per-Million Opportunities, (DPM, or DPMO). The number of errors or defects per million products or tests is a measure of the performance of a laboratory. Sigma metrics are being adopted as a universal measure of quality, and we can measure the performance of testing processes and service provision using sigma metrics 37

Sigma metrics as a quality indicator 38

Sigma Metrics A simple technique that can be used in the laboratories is to translate the method validation results into sigma metrics. Performance is characterized on a sigma scale, just as evaluating defects per million; values range from 2 to 6, where “state of the art” quality is 6 or more. In industries, 3 Sigma is considered the minimal acceptable performance for a process. When performance falls below 3 Sigma, the process is considered to be essentially unstable and unacceptable. 39

Sigma Metrics For Medical laboratories: In terms of Six Sigma performance, if a method has a value less than three, that method is considered to be unreliable and should not be used for routine test purposes. A method with low sigma levels would likely cost a laboratory a lot of time, effort, and money to maintain the quality of test results. 40

Sigma Metrics Measure Outcome Measure Variation 41

Sigma Metrics of Common Processes (US) and Laboratory Processes (Italy) 42

Sigma Metric for Process Performance In the laboratory, counting defects is also the usual Six Sigma metric technique. For example, turn around time (TAT) is very easy to define for laboratories. A laboratory might set a target of returning test results within 60 minutes of specimen receipt. Thus, when a test result is returned after 61 minutes, it’s simple to detect the defect (i. e. , TAT is > 60 minutes). Counting the number of defective test results (> 60 minutes) over a period of time is an easy way to determine the Sigma performance of the laboratory’s TAT. 43

Sigma Table 44

Sigma Metrics Calculations Controls are a known value, so variation of an observed test result can be measured. With multiple control results, information on the standard deviation of testing processes can be collected and the imprecision (coefficient of variation, % CV) can be calculated. Information about the inaccuracy (bias) of an analytical testing process can readily be calculated by comparing results between the testing method and a reference method, or by analyzing the results of the testing method in proficiency testing, peer group, or some other form of external quality assurance program. 45

Sigma Metrics Calculations All that is necessary is to decide the quality goals and calculate the method’s imprecision (CV, coefficient of variation) and bias levels as one would ordinarily do in method validation studies. Then, using the formula below, the sigma level of the method in question can readily be calculated: Sigma = (TEa – bias)/CV Where: TEa: is total error allowable (quality goal), Bias: indicator of systematic errors CV (coefficient of variation): indicator of random errors 46

Sigma Metrics Calculations For example, if a method has: A bias of 2%, A CV of 2%, and TEa of 10%, The sigma value will be: Sigma = (TEa – bias)/CV = (10 - 2)/2 = 4 The Method Decision chart displays inaccuracy on the y- axis, imprecision on the x-axis. 47

Sigma Metrics Calculations Using a hypothetical example for total calcium, we will use the systematic error from the linear regression equation y = 0. 97 x + 0. 2 An imprecision of 2%. The TEa for calcium is 1 mg/d. L. A medical decision level (where the physician might considered additional tests) for total calcium is 10 mg/d. L. Entering 10 mg/d. L into the regression equation (y = 0. 91*10 + 0. 2) gives a systematic error of 0. 7 mg/d. L; this is 70% of the total allowable error (1 mg/d. L). 48

Sigma Metrics Calculations An imprecision of 2% at 10 mg/d. L is 0. 2 mg/d. L; this is 20% of 1 mg/d. L total EA. Plotting this decision point (1) on the chart corresponds to a 2 Sigma. This is considered poor performance and would require the maximum number control measurements and several multi-rules to detect the high number of errors; many rules are needed because it would only require a small shift in the mean to result in a failure 49

Sigma Metrics Calculations If the bias could be eliminated (as demonstrated with point (2), the performance would fall into the 5 sigma range, which is considered to be very good; The performance of test for total calcium with (1) and without (2) systematic error as analyzed by a Six Sigma chart. 50

Lean Concept A lean organization focuses on creating processes that need less space, less capital, less time, and less human effort by reducing and eliminating waste. By “waste, ” we mean anything that adds no value to the process. Re-done tasks, transportation of samples, inventory, waiting, and underused knowledge are examples of waste. Lean consultants start by observing how things work currently, and they then think about how things can work faster. 51

Red segments represent failures or undesired results. In an imperfect process, there are unnecessary steps (waste), which delay results. In the 52

Lean Concept They inspect the entire process from start to finish and plan where improvements are needed and what innovations can be made in the future. Finally, they subject this to a second analysis to find ways to improve the process. Lean projects can generate dramatic reductions in turnaround times as well as savings in staffing and costs. It is said that ‘Time is money. ’ However, in laboratory medicine, time is not only money. Apart from correct test results, nothing in the laboratory is more valuable than rapid test results. 53

Lean Concept The turnaround time of the tests is crucial to decision making, diagnoses, and the earlier discharge of patients. Although Six Sigma, and the lean concept look somewhat different, each approach offers different advantages, and they do complement each other. The combination of Lean with Six Sigma is critical to assure the desirable quality in laboratory medicine for patients benefit and safety. Taken together, Lean Six Sigma combines the two most important improvement trends in quality science: making work better (using Six Sigma principles) and making work faster (using Lean Principles) 54