Validation Verification Chapter 10 1 VALIDATION VERIFICATION Very

Validation & Verification Chapter 10 1

VALIDATION & VERIFICATION • Very Difficult • Very Important • Conceptually distinct, but performed simultaneously • You must be sure your model is correct • Your client must be confident that your model is correct • Should be an integral part of model building 2

VALIDATION • Goals – Produce a model that represents system behavior closely enough to be a substitute for the system for experimentation • Analyzing & predicting performance – Increase credibility of model to managers & decision makers 3

DEFINITIONS • Verification – Ensures that the model developed is correctly implemented in the software • Validation – Ensures that the model accurately represents the realworld system 4

Validation & Verification Process • An integral part of model design & implementation process – not separate • Most methods are informal or heuristic in nature • Experience in model development, simulation programming, & the system are essential 5

MODEL BUILDING 1. Observation of real system 1. Collect data 2. Talk to members of system 2. Construct conceptual model 1. Assumptions about components & structure of system 2. Hypothesis – values of input parameters 3. Implement operational model 1. Usually using simulation software **Not linear process, iterative!! 6

SUGGESTIONS FOR VERIFICATION 1. Operational model checked by simulation software expert – not developer 2. Flow diagram for each possible action 3. Examine output for reasonableness under various inputs – use wide variety of output statistics 4. Print input at end of run to ensure stability 5. DOCUMENTATION!!! 6. Ensure animation of model is correct 7

SUGGESTIONS FOR VERIFICATION 7. Use debugger of interactive run controller (IRC) – advantages 1. Can monitor simulation progress & display results 2. Can focus on single line or process 3. Can observe model components & variables 4. Can pause; reassign values 8. GUI – always recommended ** Basic Software Engineering Principles 8

Suggestion 3 • Examine output for reasonableness – Calculate expected results – Vary input – Ask users to review • Examples Suppose multiple servers & only look at throughput. Maybe too many went to one server & too few to the other. If priority queue, are they actually processed in correct order. 9

Suggestion 3 (cont’d. ) • Utilize standard output from simulation environments • Current Count & Total Count are important variables • Consider predictions – Mathematical (e. g. Utilization) – Experts • Perform a Trace – Snapshots – By hand 10

CALIBRATION & VALIDATION • Validation – comparing model to system • Calibration – iterative process of comparing model to real system & adjusting the model – repeat! • Comparisons – Subjective – experts review – Objective – use of data & results 11

VALIDATION • Never truly completely validated – Model never totally represents the real system • Be sure model is not “fit” to one set of data 12

3 -Step Approach to Validation Naylor & Finger [1967] 1. Build a model with high face validity 2. Validate model assumptions 3. Compare model I/O transformations to corresponding I/O transformations for the real system 13

Step 1. FACE VALIDITY • Construct a model that seems valid to the users/experts knowledgeable with system • Include users in calibration – builds perceived credibility • Sensitivity Analysis – change one or more input value & examine change in results – Are results consistent with real system? – Choose most critical variables to reduce cost of experimentation 14

Step 2. Validation of Assumptions • 2 categories of assumptions – Structural assumptions – Data Assumptions • Structural Assumptions – Involves how system operates – Includes simplifications & abstractions of reality • Data Assumptions – Based on data collection & statistical analysis 15

Step 2. Validation of Assumptions (contd) • Review – Analysis of Data – Identify probability distribution – Estimate parameters of distribution – Perform goodness-of-fit test • Chi Square, Kolmogorov-Smirnov tests • Test homogeneity of data – Are two independently collected sets of data come from the same parent population? 16

Step 3. Validating I/O Transformations • Ultimate Test of a Model – Ability to predict the future behavior of the real system • Model viewed as an I/O Transformation • Can also us historical data to test prediction ability • Note: If main purpose of simulation changes, model should be revalidated 17

Step 3. Validating (cont’d) • Models are often used for comparing alternate system designs – Minor changes in parameters • IA rate, # servers – Minor change in statistical distribution – Major change in logical structure of subsystem • Queue discipline – Major design change • Manual to automated system 18

Using Historical Input Data • An alternative to randomly generated data – don’t mix different data sets • File, Spreadsheet, or Database – {A 1, A 2, …, An} & {S 1, S 2, …Sn} – Feed data into the FEL • Compare output to what happened in the real system • May be able to use technology to collect historical data for use 19

I/O Validation – Turing Test • What is the Turing Test? • Generate 5 “fake” reports from simulation & mix with 5 real reports; ask experts if they can distinguish fake from real • If cannot, then pass Turing Test! 20

Validation Techniques In order of cost-to-value ratio – Van Horn (1969, 1971) 1. Develop model with high face validity by including experts, previous research, studies, observation, experience 2. Test input data for homogeneity, randomness, goodness-of-fit 3. Turing test – use knowledgeable people 21

Validation Techniques (cont’d) 4. Compare model & system output using statistical tests 5. After model development, collect new data & repeat steps 2 to 4 6. Build new system or redesign old one, collect data on new system & use to validate model (not recommended) 7. Do little or no validation. Implement. (not recommended) 22

Conclusion • Do not become obsessed with validation & verification to the detriment of progress; causing excessive cost • Modeler should select techniques most valuable and appropriate to the particular system being modeled and the goals of the project • Validate & Verify to assure Accuracy & Credibility 23