Method Validation Studies Its a Jungle out There

Method Validation Studies It’s a Jungle out There! r

r Well Duh. . .

Purposes of Method Validation Studies: • To quantifiably characterize system performance • To assess potential for error • To identify method-to-method differences • To meet regulatory guidelines r

r When to Do Method Validation Studies: • When considering purchasing a new system • When placing a new system into service • At regular intervals to assess on-going system performance • When troubleshooting questionable system performance

Basic Components of Method Validation Studies: Any or All of the Following. . . • Precision Studies • Accuracy Studies • Range Validation/Verification Studies • Reportable Range • Critical Limits • Analytical Measurement Range • Clinically Reportable Range • Calibration Verification / Linearity • Reference Range Studies r

How to Do Method Validation Studies: r Don’t wing it!!! Consult appropriate reference guides. • NCCLS evaluation document series • Manufacturer’s package inserts, evaluation protocol manuals and supplemental protocols.

r Precision Studies • Characterize the reproducibility of a test system • Performed by repetitive testing of the sample • Intra-run • Inter-instrument • Statistical parameters • Mean • Standard Deviation (SD), mg/d. L • Coefficient of Variation (CV), % • Acceptability Expectations…the lower the better

r Accuracy Studies • Method-to-method studies comparing a POC instrument to a properly calibrated primary Laboratory analyzer. • For the purpose of assessing accuracy, the samples should be tested on both the Laboratory analyzer and the Point-of-Care instrument.

Accuracy Studies, cont’d: r • A typical Accuracy Study includes: • • all, or a representative sub-set, of Point-of-Care instruments optimally at least 40 different samples (minimum of 20 for statistical validity) good distribution of sample values ranging from low to high same samples tested on Point-of-Care instruments and Laboratory analyzer. If capillary samples are compared to venous glucose results, use only samples from fasting patients (Capillary glucose levels can be 20 -70 mg/d. L higher than venous levels at the same time in the non-fasting state).

Accuracy Studies, cont’d • Statistical analysis of accuracy study results typically includes: • Regression analysis • Bias analysis r

Accuracy Studies, Cont’d r

Regression Analysis Plotting the Comparative Data. . . r

Regression Analysis Plotting the Comparative Data. . . r

Regression Analysis Plotting the Comparative Data. . . r

Accuracy Studies, cont’d r • Regression Statistics Review: • Correlation Coefficient (r) - characterizes the dispersion of results around the line of best fit. • Slope - The “lean” of the line of best fit (proportional bias) • Y-Intercept - the point at which the line of best fit intersects the Y axis. (constant bias) • Acceptability Criteria: • Correlation Coefficient (r) - the closer to 1. 0 the better • Slope - The closer to 1. 0 the better • Y-Intercept - the closer to zero the better

r Accuracy Studies, Cont’d The Equation of the Line of Best Fit: y = mx + b m = Slope b = Y Intercept

r Accuracy Studies, Cont’d The Equation of the Line of Best Fit y = 0. 991 x + 8. 4 Slope Y-intercept

r Accuracy Studies, Cont’d The Equation of the Line of Best Fit y = 0. 991 x + 8. 4 Slope Y-intercept

Accuracy Studies, cont’d: r • Bias Analysis: • Calculate mg/d. L or % difference between Point-of-Care instrument and Laboratory analyzer for each set of paired data-points. • Acceptability Criteria (Glucose): • 95% of meter results within +/- 15 mg/d. L of Laboratory analyzer results below 75 mg/d. L or within +/-20% of Laboratory analyzer results equal to or greater than 75 mg/d. L (NCCLS document C 30 A-2)

r Bias Analysis, Cont’d Bias Chart

Range Validation Studies r • Definitions • Reportable Range - The range of results for which a system has been proven to yield numerically accurate results • Analytical Measurement Range (AMR) - The range of results through which a method yields numerical values • Clinically reportable range - (CRR) An extension of the AMR supplemented by dilution or concentration protocols • Calibration Verification - The process of verifying that a system is properly calibrated • Critical Limits - Low and high result limits which, when exceeded, require follow up action • Reference Range - Normal range

Range Validation Studies r • Definitions • Analytical Measurement Range (AMR) - The range of results through which a method yields numerical values • Clinically reportable range - (CRR) An extension of the AMR supplemented by dilution or concentration protocols • Calibration Verification - The process of verifying that a system is properly calibrated • Reportable Range - The range of results for which a system has been proven to yield numerically accurate results • Critical Limits - Low and high result limits which, when exceeded, require follow up action • Reference Range - Normal range

Range Validation Studies, Cont’d r • Validation of Reportable Range, Analytical Measurement Range, Reference Range, Critical Limits and Calibration Verification (CAP POC July 2003 Checklist, Page 34 Commentary) • Can all be accomplished through the same protocol if the range of testing spans the broadest range (AMR) • Materials used must be “matrix appropriate” such as: • Split blood samples • linearity/calibration verification materials supplied by manufacturer or other vendor • However. . .

CAP POC. 08700 - r If the laboratory has more than one method-system for performing tests for a given analyte, are they checked against each other at least twice a year for correlation of patient results? NOTE: This includes same or different instrument makes/models. This comparison must include all instruments. If multiple methods are used for a particular analyte, comparing a subset of each method to one another is acceptable only after all like instruments have been compared to one another to ensure correlation and establish that a representative subset is used in the method-to-method correlation.

r CAP POC. 08700, Cont’d: Significant changes to the commentary!!! • “POCT glucose devices are a special case in which a large number of devices may be in use in an institution. In this case alternate approaches are acceptable for documenting correlation of patient results among the devices and with the main laboratory. One approach is to document for a lot of reagent strips the agreement between patient results for a laboratory method and representative POCT device(s) and simultaneously collect results for QC materials run on the POCT devices. “ • “Other correlation schemes may be suitable. The critical factor is to have a mechanism that documents the agreement between patient results performed by the POCT glucose devices and the main laboratory. ”

r Where to Find the Method Validation Regulations CLIA CAP JCAHO

Regulatory Guidelines for Method Validation: r • CLIA 88 Final Rule - (January 24, 2003) • CAP General Checklist (July 2003) • CAP Point-of-Care Checklist (July 2003) • JCAHO - WT and QC Section (2004 Pre-Publication with Crosswalks from 2003 Guidelines) • NCCLS - C 30 A-2 - (Item 6. 3. 2).

r CLIA Guidelines for Method Validation: • CLIA 88 Final Rule - Subpart K (Quality Systems for Non-waived Tests) 493. 1253 • Revised January 24, 2003 • Outlines requirements for unmodified “non-waived”, FDA cleared tests • By definition, waived tests are not subject to items in Subpart K

CAP Guidelines for Method Validation: • CAP Laboratory General Checklist • Revised July 2003 (GEN. 42020 - GEN. 42163) • Does not distinguish based on CLIA designated test complexity • Requires validation of precision (intra- and inter-run) and accuracy • Requires verification of analytic sensitivity (Low end) • Requires assessment of analytical interference • Requires verification of reportable range (AMR) • Requires verification of reference range r

CAP Guidelines for Method Validation: • CAP Laboratory Point-of-Care Checklist • Revised July 2003 (POC. 08200 - POC. 08700) • Does not distinguish based on CLIA designated test complexity • Requires calibration verification • Requires validation of Analytical Measurement Range • Requires method-to-method and instrument-to-instrument correlation r

JCAHO Guidelines for Method Validation: r • WT Section (2004 Pre-Publication with Crosswalk from 2003 Guidelines) • Item WT. 1. 40 - Non-specific reference to requiring “Equipment performance evaluation” • QC Section (2004 Pre-Publication with Crosswalk from 2003 Guidelines) • Items QC. 1. 70 - Method Validation requirements for precision, accuracy and reportable range apply to FDA cleared, unmodified moderate complexity tests. • Requires reference range verification for all new methods

JCAHO Guidelines for Method Validation: r • QC Section (2004 Pre-Publication with Crosswalk from 2003 Guidelines) • Item QC. 1. 70 • Method Validation requirements for precision, accuracy and reportable range apply to FDA cleared, unmodified moderate complexity tests. • Requires reference range verification for all new methods • Item QC. 1. 80 • Method-to-method and instrument-to-instrument requirement specifies that waived tests are exempt.

r Informative Regulatory Links • http: //www. phppo. cdc. gov/clia/regs 2/toc. asp (CLIA Regs with 1/24/03 updates) • http: //www. aacc. org/meetings/qc/ (Oct. 29 th Teleconference • http: //www. jcaho. org/accredited+organizations/laboratory+services/standards/new+stan dards/2004+standards. htm (JCAHO Pre-publication 2004 Standards with Crosswalk) • http: //www. cap. org/html/ftpdirectory/checklistftp. html (CAP Checklist download page) • http: //www. dgrhoads. com/ee 5. sht (EP Evaluator Release 5)

r Links to Informative Articles • http: //www. cap. org/captoday/archive/2003/0303/language_barrier. html (CAP Today on AMR/CRR and Calibration Verification) • http: //www. westgard. com/cliafinalrule 5. htm (CLIA 1/24/03 method validation changes by Sharon Ehrmyeyer, Ph. D) • http: //www. westgard. com/essay 50. htm (CLIA Key 1/24/03 Changes by Dr Westgard) • http: //www. westgard. com/essay 15. htm (Method Validations: The Inner, Hidden, Deeper, Secret Meaning by Dr Westgard)

r Method Validation Studies Questions and Discussion

r