Keynote Tutorial Model quality testing and validation Keynote

  • Slides: 57
Download presentation
Keynote Tutorial: Model quality, testing and validation Keynote Tutorial: Methods for Testing and Validation

Keynote Tutorial: Model quality, testing and validation Keynote Tutorial: Methods for Testing and Validation of Simulation Models for Engineering Applications David J. Murray-Smith, School of Engineering, Rankine Building, University of Glasgow, Scotland, U. K. E-mail: david. murray-smith@glasgow. ac. uk 0

Keynote Tutorial: Model quality, testing and validation 1 The presentation 1. Introduction 2. Model

Keynote Tutorial: Model quality, testing and validation 1 The presentation 1. Introduction 2. Model testing and validation issues Uncertainties, modelling errors, testing, verification and validation. Model quality measures and model improvements. 3. Model management Libraries of sub-models and development of generic models. Model version control and model documentation. 4. Educational implications 5. Examples External validation and upgrading of linear and nonlinear helicopter models for handling qualities and flight control studies. 6. Discussion and Conclusions

Keynote Tutorial: Model quality, testing and validation 2 The purpose of models in engineering

Keynote Tutorial: Model quality, testing and validation 2 The purpose of models in engineering • To provide a basis for design. • To assist in human decision making. • To explain complex system behaviour. • For use within fault detection systems etc. . • For simulator development (e. g. for operator training or for engineering development applications).

Keynote Tutorial: Model quality, testing and validation 3 Design benefits with fit-for-purpose models 1.

Keynote Tutorial: Model quality, testing and validation 3 Design benefits with fit-for-purpose models 1. Conceptual models allow investigation of performance limitations at an early stage of the design process for normal and abnormal operating conditions. 2. Fully developed and proven models can provide information about key parameter sensitivities and inter-dependencies – useful for design decisions and optimisation. 3. Full models allow virtual prototypes to be created before any hardware prototype is available so identifying necessary design alterations at an early stage, avoiding expensive changes later on.

Keynote Tutorial: Model quality, testing and validation 4 Models in design: some pointers Ø

Keynote Tutorial: Model quality, testing and validation 4 Models in design: some pointers Ø “Improved modelling of physical and manufacturing processes will improve our ability to predict the behaviour, costs and risks of future systems, and dramatically reduce the development timescale”. UK Office of Science and Technology, Technology Foresight Panel Report (1995). Ø “Verification, validation and accreditation” (VV&A). Ø “Smart procurement” methods. Ø Concept of “the model as a specification” (as promoted by T. S. Ericsen, US Office of Naval Research).

Keynote Tutorial: Model quality, testing and validation 5 Levels of model quality Ø Level

Keynote Tutorial: Model quality, testing and validation 5 Levels of model quality Ø Level of model quality necessary is of critical importance for any given application. . . . an inappropriate model is less than useless as it may delay the project and lead to cost escalation Ø Balance needs to be found between model accuracy and the cost of developing the model. Ø Rigorous consideration of model quality is most common in applications involving safety critical issues (e. g. aeronautical engineering, automotive engineering etc. )

Keynote Tutorial: Model quality, testing and validation 6 Current problems with models q Models

Keynote Tutorial: Model quality, testing and validation 6 Current problems with models q Models are often used with very little systematic testing. q Model documentation is often minimal and is not recognised as a vital part of the model development process. q In many organisations models are passed from project to project and end up being used in ways that were never intended by the original developer of the model.

Keynote Tutorial: Model quality, testing and validation 7 Example: Hydro-turbine system modelling. Figure from

Keynote Tutorial: Model quality, testing and validation 7 Example: Hydro-turbine system modelling. Figure from Bryce, G. W. , Foord, T. R. , Murray-Smith , D. J. and Agnew, P. , ‘Hybrid simulation of water turbine governors’, Simulation Councils Proceedings, 6(1), 35 -44, 1976

Keynote Tutorial: Model quality, testing and validation 8 Photographs: © D. J. Murray-Smith

Keynote Tutorial: Model quality, testing and validation 8 Photographs: © D. J. Murray-Smith

Keynote Tutorial: Model quality, testing and validation 9 Pipeline geometry Figure from Bryce, G.

Keynote Tutorial: Model quality, testing and validation 9 Pipeline geometry Figure from Bryce, G. W. , Foord, T. R. , Murray-Smith , D. J. and Agnew, P. , ‘Hybrid simulation of water turbine governors’, Simulation Councils Proceedings, 6(1), 35 -44, 1976

Keynote Tutorial: Model quality, testing and validation First simplification Figure from Bryce, G. W.

Keynote Tutorial: Model quality, testing and validation First simplification Figure from Bryce, G. W. , Foord, T. R. , Murray-Smith , D. J. and Agnew, P. , ‘Hybrid simulation of water turbine governors’, Simulation Councils Proceedings, 6(1), 35 -44, 1976 10

Keynote Tutorial: Model quality, testing and validation Model comparisons (((Real-time approximation: continuous line Non-real-time

Keynote Tutorial: Model quality, testing and validation Model comparisons (((Real-time approximation: continuous line Non-real-time model: dashed line) Figure from Bryce, Fo Murray-Smith and Agnew, Simulation Councils Proceedings, Vol. 6, No. 1, 35 -44. Figure from Bryce, G. W. , Foord, T. R. , Murray-Smith , D. J. and Agnew, P. , ‘Hybrid simulation of water turbine governors’, Simulation Councils Proceedings, 6(1), 35 -44, 1976 11

Keynote Tutorial: Model quality, testing and validation 12 Second approximation F Figure from Bryce,

Keynote Tutorial: Model quality, testing and validation 12 Second approximation F Figure from Bryce, G. W. , Foord, T. R. , Murray-Smith , D. J. and Agnew, P. , ‘Hybrid simulation of water turbine governors’, Simulation Councils Proceedings, 6(1), 35 -44, 1976

Keynote Tutorial: Model quality, testing and validation Model comparisons for second approximation ( Real-time

Keynote Tutorial: Model quality, testing and validation Model comparisons for second approximation ( Real-time approximation: continuous line Non-real-time model: dashed line) Figure from Bryce, G. W. , Foord, T. R. , Murray-Smith , D. J. and Agnew, P. , ‘Hybrid simulation of water turbine governors’, Simulation Councils Proceedings, 6(1), 35 -44, 1976 13

Keynote Tutorial: Model quality, testing and validation Site test and simulation comparisons Figure from

Keynote Tutorial: Model quality, testing and validation Site test and simulation comparisons Figure from Bryce, G. W. , Foord, T. R. , Murray-Smith , D. J. and Agnew, P. , ‘Hybrid simulation of water turbine governors’, Simulation Councils Proceedings, 6(1), 35 -44, 1976 14

Keynote Tutorial: Model quality, testing and validation Model predictions 15

Keynote Tutorial: Model quality, testing and validation Model predictions 15

Keynote Tutorial: Model quality, testing and validation 16 Part 2: Issues of model testing

Keynote Tutorial: Model quality, testing and validation 16 Part 2: Issues of model testing and validation a) Model quality, uncertainties and modelling errors. b) Testing, verification and validation of models. c) External validation methods. d) Model quality measures in external validation.

Keynote Tutorial: Model quality, testing and validation 17 Testing of models: fitness-for-purpose Ø Tests

Keynote Tutorial: Model quality, testing and validation 17 Testing of models: fitness-for-purpose Ø Tests only deal with a small number of cases. Ø General statements about validity are impossible. Ø All testing must be carried out in the context of the application and especially the precise range of operating conditions for that application. Ø Should start from a well-understood case, even if much simplified; then move incrementally to testing for less certain situations for that application. Ø The more complex the model the harder the problem of quality assessment becomes: measures of model performance become harder to define and visualisation becomes more difficult.

Keynote Tutorial: Model quality, testing and validation 18 Aspects of model quality Ø Establishing

Keynote Tutorial: Model quality, testing and validation 18 Aspects of model quality Ø Establishing the useful range of the dynamic model for a specific application. Ø Estimating the limits of accuracy of the model (usually both for steady-state and transient conditions) in terms the magnitude of expected errors in model predictions.

Keynote Tutorial: Model quality, testing and validation 19 Sources of errors and uncertainties •

Keynote Tutorial: Model quality, testing and validation 19 Sources of errors and uncertainties • Incorrect modelling assumptions • Errors in a priori information (e, g, model parameter values) • Errors in numerical solutions of model equations • Errors in experimental procedures and in the measurements used for model testing.

Keynote Tutorial: Model quality, testing and validation Internal verification and external validation Ø Internal

Keynote Tutorial: Model quality, testing and validation Internal verification and external validation Ø Internal Verification – establishing that a computer –based model is consistent with the underlying mathematical model. i. e. “Is the simulation model right? ” Ø External Validation – demonstrating that a final (nonlinear)) model is adequate for the intended application. i. e. “Is it the right simulation model? ” Note that checks of identified linear models are sometimes referred to as “external verification”. 20

Keynote Tutorial: Model quality, testing and validation External validation Need to distinguish between: Ø

Keynote Tutorial: Model quality, testing and validation External validation Need to distinguish between: Ø Functional validation: where model is assessed in terms of how well it mimics input-output behaviour of the real system. Ø Physical/Theoretical validation where the model is based on theory and intermediate variables in model and system are compared. Approximations and assumptions are investigated within this process. 21

Keynote Tutorial: Model quality, testing and validation Approaches to external validation Ø Holistic approaches

Keynote Tutorial: Model quality, testing and validation Approaches to external validation Ø Holistic approaches (e. g. subjective opinion of an expert on the real system such as an operator). Ø Model component approaches (e. g. each subsystem tested independently and compared with corresponding components of the real system). 22

Keynote Tutorial: Model quality, testing and validation Methods for external validation Ø Methods involving

Keynote Tutorial: Model quality, testing and validation Methods for external validation Ø Methods involving direct comparisons of response data from model and real system. Ø Methods based on system identification parameter estimation. and Ø Methods involving parameter sensitivity analysis. Ø Methods based on inverse models and inverse simulations. Whatever method is used, data for external validation must be appropriate for the intended application. Careful experimental design is essential. 23

Keynote Tutorial: Model quality, testing and validation 24 Methods for system and model comparisons

Keynote Tutorial: Model quality, testing and validation 24 Methods for system and model comparisons Ø Graphical comparisons Ø Quantitative measures Ø System identification methods Ø Expert opinion Different approaches can provide different kinds of insight.

Keynote Tutorial: Model quality, testing and validation 25 Methods of system/model comparison: some examples

Keynote Tutorial: Model quality, testing and validation 25 Methods of system/model comparison: some examples Ø Plots of simulated and measured responses against an independent variable (often time). Ø Plots of simulated values against the corresponding measured values (should be 45 degree line). Ø Different graphical methods may emphasise different aspects of the simulation model performance so there are possible benefits from combining different approaches.

Keynote Tutorial: Model quality, testing and validation Typical time history comparisons 26

Keynote Tutorial: Model quality, testing and validation Typical time history comparisons 26

Keynote Tutorial: Model quality, testing and validation 27 Another time history comparison: a multi-input

Keynote Tutorial: Model quality, testing and validation 27 Another time history comparison: a multi-input multi-output case BO-105 helicopter flight test data, DLR SIMH simulation model The original version of this figure was published by the Advisory Group for Aerospace Research and Development, North Atlantic Treaty Organisation (AGARD/NATO) in AGARD Advisory Report 280 ‘Rotorcraft System Identification’, September 1991

Keynote Tutorial: Model quality, testing and validation Examples of simple cost functions Mean absolute

Keynote Tutorial: Model quality, testing and validation Examples of simple cost functions Mean absolute error Mean absolute percent error Weighted error 28

Keynote Tutorial: Model quality, testing and validation 29 Theil’s Inequality Coefficient (TIC)

Keynote Tutorial: Model quality, testing and validation 29 Theil’s Inequality Coefficient (TIC)

Keynote Tutorial: Model quality, testing and validation Use of quantitative measures: an example 30

Keynote Tutorial: Model quality, testing and validation Use of quantitative measures: an example 30

Keynote Tutorial: Model quality, testing and validation 31 Polar diagram form of display From

Keynote Tutorial: Model quality, testing and validation 31 Polar diagram form of display From Smith, M. I. , Murray-Smith, D. J. and Hickman, D. , ‘Verification and validation issues in a generic model of electro-optic sensor systems ‘ J. Defense Modeling and Simulation, 4(1), 17 -27, 2007

Keynote Tutorial: Model quality, testing and validation 32 Issues of identifiability in external validation

Keynote Tutorial: Model quality, testing and validation 32 Issues of identifiability in external validation Test input design is important since inputs must excite the system and model over an appropriate range of frequencies and amplitudes. The concept of identifiability is central to issues of test input design external validation and is thus very important for external validation. Structural identifiability relates to situations where a model may have an excess of parameters so that some specific parameters cannot be estimated uniquely for any possible experimental design (e. g. Bellman, R. and Åström, K. J. , Mathematical Biosciences, 7, 329 -339, 1970). Structural identifiability is also important for external validation.

Keynote Tutorial: Model quality, testing and validation 33 Issues of numerical identifiability and test

Keynote Tutorial: Model quality, testing and validation 33 Issues of numerical identifiability and test input design in model validation Numerical unidentifiability arises when a structurally identifiable model is being used with data that is inappropriate for the application. Numerical identifiability investigated from parameter information matrix M, the related dispersion matrix D and the parameter correlation matrix P. All depend on the sensitivity matrix X where: Inputs may maximise the overall accuracy of all parameter estimates or may be chosen to maximise accuracy of specific parameter estimates.

Keynote Tutorial: Model quality, testing and validation Upgrading of simulation models ØFollowing comparison of

Keynote Tutorial: Model quality, testing and validation Upgrading of simulation models ØFollowing comparison of model and system behaviour usually need to analyse discrepancies and propose upgrades for model. ØChanges must be evaluated systematically on a physical basis – with further iterations in the development cycle. ØParametric changes usually considered first, before structure. ØSometimes possible to associate model deficiencies with specific state variables model (e. g. correlation of output error with a state variable may help identify problem source). ØCorrelation of model errors with derivatives of state variables may suggest that a higher-order description would be more useful. ØOptimisation tools (including evolutionary computing methods such as GA and GP) may be useful but should be used along with physical knowledge and understanding of the model. 34

Keynote Tutorial: Model quality, testing and validation 35 Part 3: Model management a) Libraries

Keynote Tutorial: Model quality, testing and validation 35 Part 3: Model management a) Libraries of sub-models and generic models. b) Model version control and model documentation.

Keynote Tutorial: Model quality, testing and validation 36 Generic and re-usable sub-models Ø Generally

Keynote Tutorial: Model quality, testing and validation 36 Generic and re-usable sub-models Ø Generally accepted that system design should be based on use of generic descriptions and re-usable sub-models. Ø Examples of the generic approach may be found in automotive engineering, gas turbines etc. Issues inevitably arise in the external validation of generic models – one approach is discussed in Smith, M. I. , Murray-Smith, D. J. and Hickman, D. , ‘Verification and validation issues in a generic model of electro-optic sensor systems’ J. Defense Modeling and Simulation, 4(1), 17 -27, 2007

Keynote Tutorial: Model quality, testing and validation 37 Model documentation and version control Ø

Keynote Tutorial: Model quality, testing and validation 37 Model documentation and version control Ø Extra costs of creating good documentation should be more than balanced by the resulting re-usability of models. Version control processes should ensure that changes are fully documented. Ø Documentation and version control well developed in software engineering field. Same principles should be applied to the model development process.

Keynote Tutorial: Model quality, testing and validation Items for documentation Ø Purpose of model

Keynote Tutorial: Model quality, testing and validation Items for documentation Ø Purpose of model and the intended application. Ø Full description of model and corresponding computer code. Ø List of all assumptions and approximations used. Ø Details of all tests carried out on the real system to provide information for model development. Ø Details of the internal verification process. Ø Details of the external validation process, with statements about why model was accepted or rejected and information about usable range for model. 38

Keynote Tutorial: Model quality, testing and validation Part 4: Educational implications 39

Keynote Tutorial: Model quality, testing and validation Part 4: Educational implications 39

Keynote Tutorial: Model quality, testing and validation 40 Modelling and simulation in engineering education

Keynote Tutorial: Model quality, testing and validation 40 Modelling and simulation in engineering education Ø Engineering students encounter mathematical and computerbased modelling repeatedly in their university education. Ø Emphasis is most often on development of models from physical principles and on using models/simulations in place of experiments on real systems.

Keynote Tutorial: Model quality, testing and validation 41 Model quality and testing issues in

Keynote Tutorial: Model quality, testing and validation 41 Model quality and testing issues in engineering education Ø Issues of model quality and fitness-for-purpose are seldom emphasised in the teaching of modelling and simulation. Ø Model validation is neglected in education. The teaching of system modelling and simulation should include much more on model validation methods. Ø Model testing should become second nature for students. Ø Documentation, model re-use and libraries of models must be given much more emphasis (especially in more advanced teaching).

Keynote Tutorial: Model quality, testing and validation 42 Part 5: Examples Drawn from external

Keynote Tutorial: Model quality, testing and validation 42 Part 5: Examples Drawn from external validation and upgrading of helicopter models for flight control system design.

Keynote Tutorial: Model quality, testing and validation Model limitations in control Ø Plant model

Keynote Tutorial: Model quality, testing and validation Model limitations in control Ø Plant model limitations often impose serious performance limitations within control system, especially in systems with high-performance requirements. Ø Particularly important to have highly accurate plant models for the part of the frequency range close to the “cross-over” region in the frequency domain. 43

Keynote Tutorial: Model quality, testing and validation 44 Examples from aircraft and helicopter flight

Keynote Tutorial: Model quality, testing and validation 44 Examples from aircraft and helicopter flight control system design • There are many well documented aircraft flight control examples illustrating problems of model quality and model limitations in integrated system design. • Problems are often identified during initial testing of hardware. These lead to development of improved models and corresponding control design changes. • The later in the design cycle these changes have to be made the more costly they are and the greater the delays to the project.

Keynote Tutorial: Model quality, testing and validation 45 Helicopter model requirements Ø Must perform

Keynote Tutorial: Model quality, testing and validation 45 Helicopter model requirements Ø Must perform well over a defined range of frequencies. Ø Must perform well over a defined range of manoeuvre amplitudes. Photograph: © D. Murray-Smith ØFor flight control design model must perform especially well close to open-loop gain and phase cross-over frequencies (as with control applications involving other types of system).

Nominal model: simulation and flight data compared for same test inputs. (Westland Lynx helicopter

Nominal model: simulation and flight data compared for same test inputs. (Westland Lynx helicopter in 300 ft. quick-hop manoeuvre) From Bradley, R. , Padfield, G. D. , Murray-Smith, D. J. and Thomson, D. G. , ‘Validation of helicopter mathematical models’, Transactions of the Institute of Measurement and Control’, 12(4), 186 -196, 1990.

Common test inputs used for system identification and “external verification” of the identified model

Common test inputs used for system identification and “external verification” of the identified model (BO-105 flight data) SS The original versions of these figures were published by the Advisory Group for Aerospace Research and Development, North Atlantic Treaty Organisation (AGARD/NATO) in AGARD Advisory Report 280 ‘Rotorcraft System Identification’, September 1991

Keynote Tutorial: Model quality, testing and validation 49 . . . and flight testing

Keynote Tutorial: Model quality, testing and validation 49 . . . and flight testing

Simulation, identification and “external verification” results (BO-105 flight test data, DLR SIMH model) The

Simulation, identification and “external verification” results (BO-105 flight test data, DLR SIMH model) The original version of these figures were published by the Advisory Group for Aerospace Research and Development, North Atlantic Treaty Organisation (AGARD/NATO) in AGARD Advisory Report 280 ‘Rotorcraft System Identification’, September 1991

Estimated and theoretical parameter values of the identified model for lateral/directional characteristics (SA-330 Puma

Estimated and theoretical parameter values of the identified model for lateral/directional characteristics (SA-330 Puma helicopter flight test data : 80 knots straight and level flight) L L p N p r N v L N r v Y v L δ lat Normalised HELISTAB values 2ζω0 N N δ lat L Values estimated from flight data using system identification methods. δ ped

Keynote Tutorial: Model quality, testing and validation 53 Assessment of a theoretical nonlinear model

Keynote Tutorial: Model quality, testing and validation 53 Assessment of a theoretical nonlinear model for a Puma helicopter Parameter values for two different flight conditions, showing parametric trends from a physically-based nonlinear simulation model (HELISTAB) and the trends in estimates from flight tests involving system identification of separate linearised models for each flight condition. From Bradley, R. , Padfield, G. D. , Murray-Smith, D. J. and Thomson, D. G. , ‘Validation of helicopter mathematical models’, Transactions of the Institute of Measurement and Control’, 12(4), 186 -196, 1990.

Keynote Tutorial: Model quality, testing and validation 55 Summary of model quality issues for

Keynote Tutorial: Model quality, testing and validation 55 Summary of model quality issues for helicopter flight control system design ØGood vehicle models are essential for design of high-bandwidth full-authority active flight control systems. ØPublished examples show that the achievable performance of flight control systems have, in some cases, been overestimated in initial design studies, usually because of limitations of the flight mechanics model of the vehicle (see e. g. Tischler, M. B. Advances in Aircraft Flight Control Systems, Taylor & Francis, London 1996). ØAlthough control systems can be made robust to compensate for poor accuracy this is usually at the expense of performance. Improved modelling procedures and improved models can offer significant benefits. Otherwise, problems may not be apparent until the flight testing stage, leading to costly redesign, extended flight test programmes and delays in certification.

Keynote Tutorial: Model quality, testing and validation 56 Part 5: Discussion and conclusions

Keynote Tutorial: Model quality, testing and validation 56 Part 5: Discussion and conclusions

Keynote Tutorial: Model quality, testing and validation 57 Discussion: Model quality in design Ø

Keynote Tutorial: Model quality, testing and validation 57 Discussion: Model quality in design Ø The more demanding the design specification the more important is the fitness-for-purpose of models used in design. Ø More attention needs to be given to the external validation of models for the specific application in question. a) Establishing the useful range of the model. b) Estimating the accuracy of the model within that range. Ø Model validation is part of the model building process and external validation techniques need to be applied repeatedly. Ø Models should be retained, maintained and updated throughout the whole life-cycle of the system that they represent.

Keynote Tutorial: Model quality, testing and validation 58 Recommendations • More attention should be

Keynote Tutorial: Model quality, testing and validation 58 Recommendations • More attention should be given to the fitness-for-purpose of models used in design, especially for demanding applications. • Current methods for external validation are time-consuming and difficult to apply in many situations. More effort should be devoted to improving validation methods. • Techniques of version control and rigorous documentation should borrowed from software engineering and applied to the model development process. Re-use of proven models should be made easier and more comprehensive model documentation should be available within model libraries. • Issues of model quality should be given far more attention within engineering education.

Keynote Tutorial: Model quality, testing and validation 59 Conclusions v With suitable structure and

Keynote Tutorial: Model quality, testing and validation 59 Conclusions v With suitable structure and parameter values and rigorous external validation models that are fit-for-the-purpose of a given application can be developed (iteratively of course). v Good model management can reduce the cost of design and development. v There are no quick answers: a systematic approach is essential, moving incrementally from well-understood cases to less well known situations. v Educational and cultural changes are needed as well as improved management of the modelling, simulation and design processes within most organisations.