Useful Tips for Presenting Data and Measurement Uncertainty

Useful Tips for Presenting Data and Measurement Uncertainty Analysis Ben Smarslok University of Florida Mechanical and Aerospace Engineering 1

Outline • Why is presenting data properly important? • Explain important terminology and definitions – NIST vs. ISO vs. ASME/ASTM – Oberkampf definitions of model uncertainty (not included) • Experimental scenarios and corresponding methods – – Uncertainty propagation Crossed vs. nested factors (ANOVA vs. VCA) p-values Interlaboratory Studies (not included) University of Florida Mechanical and Aerospace Engineering 2

m = 75 ± 5 g What is the meaning of ± 5 ? • • • Best guess by experimenter Half the smallest division of measurement Standard deviation: Standard error: m = / n Expanded uncertainty of ± 2 or ± 3 (95% or 99% confidence interval) • Standard uncertainty: u • Combined standard uncertainty: uc *(Courtesy of Duane Deardorff presentation from UNC) University of Florida Mechanical and Aerospace Engineering 3

What does x ± u mean? • • • Engineers think in terms of ± 2 (95%) Physicists generally report ± 1 (68% CI) Chemists report ± 2 or ± 3 (95% or 99% CI) Survey/poll margin of error is 95% CI Accuracy tolerances are often 95% or 99% NIST Calibration certificate is usually 99% • Conclusion: The interpretation of ± u is not consistent within a field, let alone between fields – It is very important to explain the statistical relevance of the uncertainty bounds!!! University of Florida Mechanical and Aerospace Engineering 4

Presenting Uncertainty Precisely • Choose a standard for presenting uncertainty (I prefer NIST), and reference the standard • Explain the source of the uncertainty – Type A – calculated by statistical methods (it is useful to explain the design of experiments and the number of samples involved) – Type B – determined by other means, such as estimate from experience or manufacturers specifications • Use terms carefully! – Error vs. Uncertainty: Error is the deviation from the true value and measured value (never known), which is estimated as uncertainty – Bias vs. variability (will explain later) • Avoid use of ambiguous ± notation without explanation • Pet peeve: – COV = covariance University of Florida Mechanical and Aerospace Engineering 5

NIST Classification of Measurement Uncertainties • Uncertainty classification: – Random uncertainty / variability – scatter in the measurements (v) – Systematic uncertainty / bias – systematic departure from the true value (b) • Type of evaluation: – Type A – calculated by statistical methods – Type B – determined by other means, such as estimate from experience Range is at 95% (2 ) level of a normal distribution xt = true value of specimen m = experimental population average x = experimental sample average vx = random error of sample bx = systematic error of sample University of Florida Mechanical and Aerospace Engineering 6

Uncertainty Analysis Example • Consider our transverse modulus work (E 2) • Hooke’s Law: ·P = Load 2 ·A = Area 1 ·e 2 = transverse strain • We will work through this problem backwards Total Uncertainty Bias & Variability Components University of Florida Mechanical and Aerospace Engineering Contributors 7

Level 1: Total Uncertainty • In general, – where, v. X and b. X were propagated from component uncertainties = Student’s t distribution at 95% confidence level (depends on # of DOF) • Total uncertainty of E 2 at 1 (68%) confidence for comparison to experimental results • Or, at the commonly accepted 95% level University of Florida Mechanical and Aerospace Engineering 8

Level 2: Uncertainty Propagation • Law of Propagation of Uncertainties (LPU): – where, p are the inputs (components) and q is the output • E 2 Example: – Uncertainty contributors were analyzed for each of the components of E 2 – Random and systematic effects propagated separately – Only systematic uncertainties can have correlated effects • Thickness and width are correlated University of Florida Mechanical and Aerospace Engineering 9

Level 3: NIST Component Measurement Uncertainty Table University of Florida Mechanical and Aerospace Engineering 10

Level 4: Contributors of Component Uncertainty (Further Analysis) • Numerous different methods to analysis the significance of uncertainty contributors • It is important to use the appropriate analysis method depending on the design of experiments (DOE) – Either design the experiments properly or match the corresponding method to the data you already have • Most DOEs fall into one of these two categories: Crossed Same patients in each hospital. Nested Patients unique to each hospital. University of Florida Mechanical and Aerospace Engineering 11

Crossed Design: ANOVA • Crossed (or factorial) DOEs correspond to analysis of variance (ANOVA) • Consider thickness in the E 2 example – Since the SAME specimens were measured in the SAME positions with the SAME users, then the factors were crossed – 3 -way ANOVA with crossed, random variables was conducted Nominal: 0. 09 x 1 in. Uncertainty contributors: Specimen – variability from specimen to specimen Position – variation across measurement surface User – error from user technique Measurement repeatability University of Florida Mechanical and Aerospace Engineering 12

Thickness ANOVA • 3 -way ANOVA of crossed, random variables – Statistical software available for ease of use: Excel for 2 factors or SAS for 3 or more • Factors: – – A = specimen B = position C = user Repetitions: a=4 b=3 c=4 n=3 Hypothesis Test for A: • ANOVA model: • ANOVA results were not directly used in uncertainty analysis, but were used to identify significant contributors and validate uncertainty estimates University of Florida Mechanical and Aerospace Engineering 13

Results: Thickness ANOVA • Use ANOVA to deterimine the ~ ~ ~ significance of the contributors of uncertainty in thickness • Position is most significant factor with p-value = 0. 006 • Not as interested in interactions in this study • Used to validate estimated range of uncertainties of thickness and width University of Florida Mechanical and Aerospace Engineering 14

Nested Design: VCA • Nested DOEs correspond to variance component analysis (VCA) • Consider a two-stage nested design of one specimen for thickness – Relevant if positions and users were unique each time – Specimens considered individually since thickness does not have to be the same from one specimen to the next – Data was organized according to position – y 1, y 2, and y 3 refer to the repeated basic measurements University of Florida Mechanical and Aerospace Engineering 15

Variance Component Analysis of Thickness • Goal: Develop a nested design to determine the contribution of each factor in the overall variance where, i is a component in the process • Variance of the measurement process for one specimen – Position – the three locations on the specimen where thickness was measured (unique to each specimen) – User – four different users per position performed the measurements – Basic Measurement – three repeated measurements by each user at each position • Compare the weight of each contributor to determine significance University of Florida Mechanical and Aerospace Engineering 16

Concluding Remarks • Using proper statistical terminology and representation is necessary to have meaningful results • You can say your results are “pretty good”, but give what your definition of “pretty good” is! • Depending on the project, more or less uncertainty analysis may be required • It is important to design your experiments with the statistical analysis in mind • Age-old question: How many measurements do I need? – Obviously depends on the circumstances, so there is no straight forward answer – Best recommendation: Feel comfortable enough with your results that you can predict the next measurement within a desired range University of Florida Mechanical and Aerospace Engineering 17