Establishing Model Credibility Using Verification and Validation Joe

Establishing Model Credibility Using Verification and Validation Joe Hightower Sr. Quality Engineer Associate Technical Fellow The Boeing Company Phone: 425. 753. 4882 Email: joe. c. Hightower@boeing. com 6/18/2021 1

Two goals Provide an introduction to Verification and Validation in the Modelling and Simulation Context Describe the ASME relationship to Verification and Validation and available resources 6/18/2021 2

Introducing Verification and Validation For Modelling and Simulation 6/18/2021 3

Models Fashion Models Engineering Models Process Models Mathematical Models Visualize/Improve Understanding Explain/Teach/Communicate Answer Questions/Guide Decisions 6/18/2021 4

“Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful” Box and Draper “Empirical Model-Building and Response Surfaces” 6/18/2021 5

Making Decisions Uncertainty in Simulation 2. Simulation Results Probability of Decision Error 3. Risk from Error Make a Decision 1. Targets & Thresholds Cost or Impact of Error 6/18/2021 6

What is Uncertainty? uncertainty: a potential deficiency in any phase or activity of the modeling, computation, or experimentation process that is due to inherent variability or lack of knowledge. V&V 10 “Guide for Verification and Validation in Computational Solid Mechanics” 6/18/2021 7

Model Building Process Conceptual Models • Descriptions and pictures • Model form errors Mathematical Models • Equations relating inputs to outputs Encoded Models • Convert Math into Executables Simulate • Ask and answer Questions 6/18/2021 • Mistakes in translating the model into code • Errors in numerical approximation • Errors in inputs • Floating point errors 8

Verification and Validation Verification Validation 6/18/2021 • Solving the Equation Right • Mistakes in translating the model into code • Errors in numerical approximation • Solving the Right Equation • Errors in inputs • Floating point errors • Model form errors 9

ASME and Verification & Validation Leading the effort to develop V&V methodologies 6/18/2021 10

ASME VISION STATEMENT: To be the world leader in mechanical and multidisciplinary engineering codes, standards, conformity assessment programs, and related products and services. MISSION STATEMENT: Develop the preeminent, universally applicable codes, standards, conformity assessment programs, and related products and services for the betterment of humanity. Involve the best and brightest people from around the world to develop, maintain, promote, and employ ASME products and services globally. 6/18/2021 11

ASME Activity In V&V Past 30 years leading the development of methods for assessing measurement uncertainty • Current Standard is ASME PTC 19. 1 -2005 Test Uncertainty Standard 6/18/2021 Past 20+ years leading the development of methods for code and solution verification and validation • Numerical uncertainty quantification • Simulation Validation • Three currentin Resulting Standards • Verification and Validation Symposium • Verification and Validation Journal 12

Current Publications 6/18/2021 13

V&V Symposium The first large-scale symposium dedicated entirely to Verification, Validation, and Uncertainty Quantification of Computer Simulations Multidisciplinary discussion and exchange of ideas and methods for Verification and Validation in Computer Simulations Held every May; 2017 Meeting is scheduled for May 3 -5, 2017 6/18/2021 14

V&V Journal • Launched 2016 • V&V Committee serves in advisory role to Journal • Has published 4 Issues to date 6/18/2021 15

V&V 30 Verification and Validation in Computational Simulation of Nuclear System Thermal Fluids Behavior V&V 60 Verification and Validation in Modeling and Simulation in Energy Systems and Applications V&V 40 Verification and Validation in Computational Modeling of Medical Devices V&V 50 Verification and Validation of Computational Modeling for Advanced Manufacturing V&V 20 Verification and Validation in Computational Fluid Dynamics and Heat Transfer V&V 10 Verification and Validation in Computational Solid Mechanics 6/18/2021 Cross. Computational Industry Problems Applications Verification & Validation Standards Committee Technology Specific ASME Structure 16

ASME V&V 50 To provide procedures for verification, validation, and uncertainty quantification in modeling and computational simulation for advanced manufacturing. * *Advanced Manufacturing, as defined in the PCAST report: “Advanced manufacturing is a family of activities that (a) depend on the use and coordination of information, automation, computation, software, sensing, and networking, and/or (b) make use of cutting edge materials and emerging capabilities enabled by the physical and biological sciences, for example nanotechnology, chemistry, and biology. It involves both new ways to manufacture existing products, and the manufacture of new products emerging from new advanced technologies. ” —President’s Council of Advisors on Science and Technology Report to the President on Ensuring American Leadership in Advanced Manufacturing - 2011 6/18/2021 17

V&V 50 Scope • New Technologies • Existing Technologies Advanced Manufacturing • Data Driven • Physics based • Hybrid 6/18/2021 V&V 50 Working Groups Model Types Areas of Application • Process Development • Process Control • Quality Control 18

V&V 50 working groups Terminology, Concepts, Relationships and Taxonomy for VVUQ in Advanced Manufacturing VVUQ Applications in Process Technologies V&V 50 Working Groups VVUQ Methods in Data-driven and Hybrid Models 6/18/2021 VVUQ Challenges and Methods in Systems of Models V&V Interactions with the Model Life Cycle 19

ASME - INCOSE interface Study the framework developed and adapt to other modelling problems Use current standards where appropriate for the problem 6/18/2021 Engage in relevant standards development Potential INCOSE interactions with ASME Potential opportunities for Codevelopment for other modeling problems relevant to Systems Engineering 20