Validating a Diagnostic Device Using Gage RR and

Validating a Diagnostic Device Using Gage R&R and DOE Cheryl Pammer Minitab Inc. © 2016 Minitab, Inc.

Agenda Through the use of a customer success story, I will illustrate the successful use of Gage R&R, DOE and Tolerance Interval techniques for demonstrating the capability of a diagnostic test machine. Tools used in the analysis: ► A Gage R&R study for a destructive measurement ► A DOE using a Central Composite Design ► Analysis of both mean and variation of response data © 2016 Minitab, Inc.

Non-disclosure Statement All results and scenarios are based on the authors’ actual experiences. Data units, variable names, etc. have been changed for confidentiality reasons. © 2016 Minitab, Inc.

Gage R&R with Destructive Test • Machine automates the analysis of chemical components in urine used to diagnose various medical conditions. • The goal of this study is to determine whether the variation in glucose measurements obtained from this machine is acceptable. © 2016 Minitab, Inc.

Gage R&R Crossed Overall Variation Part-to-Part Variation Measurement System Variation Repeatability Variation due to operator Reproducibility Operator © 2016 Minitab, Inc. Operator*Part

Standard vs Destructive Gage R&R Standard Study The only way to measure true repeatability is through repeats. Destructive Test These are not repeats! © 2016 Minitab, Inc.

Subsamples for Each Operator Assume these are repeat measurements. Estimate repeatability using crossed analysis. When to Use: Take enough subsamples from the original part to be tested by each operator. Limitation: Repeatability is inflated (Repeatability + Subsample-to-Subsample variation). © 2016 Minitab, Inc.

Estimating Repeatability A single urine sample is split into six subsamples. A glucose reading is taken from each subsample. The six subsamples: ► Are similar enough to represent the same part. ► Provide a measure of repeatability. © 2016 Minitab, Inc.

Sampling Plan for Study ► 20 randomly selected urine samples, each split into 6 subsamples ► 3 randomly selected operators ► Each operator measures 2 subsamples taken from each sample © 2016 Minitab, Inc.

Repeatability Using Subsamples • Subsamples can represent repeat measurements. • Assume within-sample variation is negligible. • Examples: ü Different locations on the same part. ü Subsamples from a single batch, using entire batch. ü Subsamples from a single batch, not using entire batch. © 2016 Minitab, Inc.

Repeatability Component Repeatability = σ2 Within-Gage + σ2 Subsample ► s 2 subsample is assumed negligible. ► High repeatability variation could mean subsamples are non-homogeneous. © 2016 Minitab, Inc.

Results of the Study ► Total Gage R&R variation is 22. 7%. ► The Operator By Sample interaction is the largest source of measurement system variability. © 2016 Minitab, Inc.

Sample by Operator Interaction ► The effect of operator is not the same for each sample. ► Investigate why some samples were harder to measure. © 2016 Minitab, Inc.

Conclusions ► The total Gage R&R is 22. 7%. ► 16. 12% of this variation is due to the interaction between operator and sample. AIAG Guidelines for Total Gage R&R Variation: % Contribution System is… < 1% Acceptable 1% - 9% Marginal > 9% Not Acceptable © 2016 Minitab, Inc.

Design of Experiments Use to efficiently study the effects of several factors on a process. © 2016 Minitab, Inc.

Urinalysis Results The machines automate the analysis of chemical reactions to test strips that measure: • specific gravity • p. H • protein • glucose • ketone • blood • leukocyte • nitrite • urobilinogen • bilirubin • color • clarity Some substances are known to interfere with (bias) measurements of other substances. © 2016 Minitab, Inc.

Design Objectives Objective: Study the potential interference of specific gravity, p. H, and ascorbic acid on mean and variation in glucose readings. Could interference cause the test to miss a high (1900+) glucose reading? Quadratic relationships between the response and the factors are expected. Interactions between factors are also likely. • Factors: Specific Gravity, p. H, Ascorbic Acid • Response: Glucose © 2016 Minitab, Inc.

Response Surface Designs ► Determine effect size ► Investigate interactions ► Fit quadratic models ► Optimize settings Y = b 0 + b 1*A + b 2*B + b 3*A*B + b 4*A 2 + b 5*B 2 + e © 2016 Minitab, Inc.

Central Composite Designs Include: ► Corner points ► Axial points ► Center points © 2016 Minitab, Inc.

Data Collection Plan A single, spiked urine sample with a specific setting of SG, p. H, and AA and glucose of 1900 is split into 6 subsamples. Response: A glucose reading is taken from each subsample. • The sub-samples represent REPEAT measurements, not REPLICATES. • How do we handle repeats and replicates in the analysis? © 2016 Minitab, Inc.

Repeats vs. Replicates Replicate Repeat • Replicate variation is the variation between different samples. Typically used to assess the significance of factors. • Repeat variation is the variation within a single sample. Repeats allow the estimation of the mean and variation when the settings are held fixed. © 2016 Minitab, Inc.

Analysis of Mean and Variation Use Analyze Response Surface Design to analyze the mean and variation of the repeat measurements: ► Response 1: Mean of the repeated glucose readings ► Response 2: Natural log of the standard deviation of the repeated glucose readings © 2016 Minitab, Inc.

Analysis of Variance Results for Mean • The interaction between p. H and AA is significant • The square terms are significant indicating quadratic relationships © 2016 Minitab, Inc.

Effects of SG, p. H, and AA on the Mean The setting at which the interference decreases the glucose reading by the largest amount is: • Specific Gravity = 1. 02 • p. H = 8 • Ascorbic Acid = 100 The mean glucose reading at this worst setting is 1828. 11. © 2016 Minitab, Inc.

Effects of SG, p. H, and AA on Variation Analyze Response Surface Design for the Ln. Std. Dev response produced no significant effects – repeat variation remained constant across settings. © 2016 Minitab, Inc.

Worst-case Interference ► The mean glucose reading is 1828. 11 at the worst setting ► The machine will classify readings into color blocks: ► From past experimentation, the cutoff point at which most samples will fall into the 1900+ color block is 1750. ► Can we be confident that interference would not cause a reading for a sample with true glucose of 1900 to fall below the 1750 cutoff? © 2016 Minitab, Inc.

Conclusions Follow-up experimentation at the worst settings of specific gravity, p. H, and ascorbic acid showed that machine was not likely to miss a high (1900+) glucose reading. © 2016 Minitab, Inc.

Questions? Cheryl Pammer cpammer@minitab. com Dwayne Walker dwalker@minitab. com © 2016 Minitab, Inc.