Verification and Validation of Agentbased Scientific Simulations Xiaorong

Verification and Validation of Agent-based Scientific Simulations Xiaorong Xiang, Ryan Kennedy, Gregory Madey Computer Science and Engineering University of Notre Dame Steve Cabaniss Department of Chemistry University of New Mexico 1/6/2022 1

Overview p Introduction n n Concepts of Verification and Validation Research Objectives and Methods A Case Study p Apply Verification and Validation Methods to the Case Study p Conclusion p Future Work p 1/6/2022 2

Model Verification & Validation (V & V) p V&V n n p Verification: get model right Validation: get right model The cost and value influence confidence of model acceptance level *Adapted from Sargent: “Verification and Validation of Simulation Models” 1/6/2022 3

V & V for Agent-based Simulation p p Agent-based modeling is a new approach Different than Queuing Models n n n p Entities: large number of heterogeneous active objects vs. passive objects Space: continuous or discrete grid space vs. network of servers and queues Interactivity: high vs. low Active components: agents vs. queues and servers Goal: discovery vs. design and optimization Few literature to date address the formalized methodology for V & V of Agent-based Simulations 1/6/2022 4

What and How p Research objective n p Generate guidelines or a formalized methodology for V & V of Agent-based Simulations How n n n 1/6/2022 NOM project as a case study Evaluate and adapt the formalized V & V techniques in industrial and system engineering for DES Identify a subset of these techniques that are more cost-effective for Agent-based Simulations 5

NOM Agent-based Simulation Model p NSF funded interdisciplinary project n n n Understanding the evolution and heterogeneous structure of Natural Organic Matter (NOM) E-science example Chemists, biologists, ecologists, and computer scientists Agent-based stochastic model p Web-based simulation model p 1/6/2022 6

NOM p What is NOM? n p Heterogeneous mixture of molecules in terrestrial and aquatic ecosystems Why study NOM? n n 1/6/2022 Plays a crucial role in the evolution of soils, the transport of pollutants, and the global carbon cycle Understanding NOM helps us better understand natural ecosystems 7

The Conceptual Model I p Agents n A large number of molecules p Heterogeneous properties § Elemental composition § Molecular weight § Characteristic functional groups n Behaviors Transport through soil pores (spatial mobility) p Chemical reactions: first order and second order p Sorption p 1/6/2022 8

Stochastic Synthesis: Data Model Pseudo-Molecule Elemental Functional Structural Composition 1/6/2022 Calculated Chemical Properties and Reactivity Location Origin State 9

The Conceptual Model II p Stochastic Model n Individual behaviors and interactions are stochastically determined by: p Internal attributes § Molecular structure § State (adsorbed, desorbed, reacted, etc. ) p External conditions § Environment (p. H, light intensity, etc. ) § Proximity to other molecules p p Space n p Length of time step, Δt 2 D Grid Structure Emergent properties n 1/6/2022 Distribution of molecular properties over time 10

Implementations 1/6/2022 11

V & V of the NOM Model p Examples of V & V techniques n Face validity p p n n n n 1/6/2022 Animation Graphical representation Tracing Internal validity Historical data validation (calibration sets and test sets) Sensitivity analysis Prediction validation Comparison with other models Turing test 12

V & V of NOM Simulation Model 1/6/2022 *Adapted from Sargent: “Verification and Validation of Simulation Models” 13

Face Validity 1/6/2022 14

Internal Validity I 1/6/2022 15

Internal Validity II 1/6/2022 16

Model-to-Model Comparison I p p p Compare the model with validated one Compare the model with non-validated one Different implementations n n p Different programming languages Different packages Different modeling approaches n Predator-Prey model p p Agent-based approach vs. System Dynamics approach Powerful method for ABS 1/6/2022 17

Model-to-Model Comparison II Features Alpha Step No-flow Reaction Developing Group University of New Mexico, chemists University of Notre Dame, computer scientists Programming language Pascal Java (Sun JDK 1. 4. 2) Platforms Delphi 6, Windows Red hat Linux cluster Running mode Standalone Web based, standalone Simulation package None Swarm, Repast libraries Animation None Yes Spatial representation None 2 D grid Second order reaction Random pick one from list Choose the nearest neighbor First order with split Add to list Find empty cell nearby 1/6/2022 18

Model-to-Model Comparison III 1/6/2022 19

Model-to-Model Comparison IV 1/6/2022 20

Model-to-Model Comparison V 1/6/2022 21

Conclusion and Future Work V & V Case Study p Model-to-Model Comparison is Powerful p Collect and evaluate more statistical data p Compare simulation results against empirical data p Tweak V & V methods p Generate guidelines and methodology for V & V of agent-based simulation models p 1/6/2022 22

Questions or Comments? 1/6/2022 23