ModelValidation in ModelBased Development Kurt Woodham L3 Communications
Model-Validation in Model-Based Development Kurt Woodham L-3 Communications OSMA SAS ’ 08 SAS_08_Model_Val_Exec_Heimdahl Ajitha Rajan, Mats Heimdahl University of Minnesota September 8 -12 MAC-T IVV-08 -151
Problem: Model Validation • Model-Based Development (MBD) is here to stay u Use of MBD is accelerating n n u Enhances early detection of requirement, design, or implementation defects n • • Estimate 50% of NASA development projects using some form of MBD Many advantages: model-checking, code generation, desktop testing, closed-loop simulation “Executable Specifications” enable evaluation of behavior that might otherwise be relegated to Inspections and Testing How do we know the models are “right”? u Manually develop black-box tests When have we validated enough? u Measure test coverage on an implementation/model SAS_08_Model_Val_Exec_Heimdahl 2 MAC-T IVV-08_151
Problem : Current Practice • Measure black-box test coverage over the model u Indirect measure n Defects of omission in model not exposed. Incomplete Model Weak Black-Box Test set u Executable artifact is necessary n Adequacy can only be determined late in the development process SAS_08_Model_Val_Exec_Heimdahl 3 MAC-T IVV-08_151
Goals of Project • Define metrics for objective, implementation-independent measure of adequacy of a black-box test suite • Develop tools to measure validation adequacy based on the defined metrics • Provide capability for autogeneration of black-box test suites SAS_08_Model_Val_Exec_Heimdahl 4 MAC-T IVV-08_151
Testing – What does it mean? Assertions In General Does it implement? Specification Assertion Based Testing (ABT) to Validate Model Does it implement? Implementation Model Does it implement? Model-Based Testing (MBT) to Verify Code Source Code Our contribution is in providing novel ABT capabilities SAS_08_Model_Val_Exec_Heimdahl 5 MAC-T IVV-08_151
What are Assertions? Properties/ Formal Assertions Defined over Can also be over components, interfaces, . . . in 1 out 1 System ink SAS_08_Model_Val_Exec_Heimdahl 6 outm MAC-T IVV-08_151
Contributions - ABT 2 Auto-generate Assertions Black-Box Tests Validate 3 Assess Model and Assertion Completeness Model Measure Adequacy 1 We provide the following contributions in the Assertion-Based testing domain (indicated by in the above figure): 1. Objective, implementation-independent measure of adequacy of a blackbox test suite 2. Auto-generation of black-box validation tests directly from assertions 3. Objective assessment of completeness of model as well as assertions SAS_08_Model_Val_Exec_Heimdahl 7 MAC-T IVV-08_151
Relevance to NASA • • • MBD is here - estimate one-half of all NASA missions in development or on the books will use model-based subsystem development u Extensive use in avionics industry How do we know the models are right? u Model validation problem We provide the capability to u u Objectively measure the “quality” of assertion-based black-box validation tests Objectively assess the completeness of a model n u Objectively assess the adequacy of a set of assertions n u Does the model address all assertions? Are there enough assertions to adequately describe the model? Automatically generate truly assertion-based tests SAS_08_Model_Val_Exec_Heimdahl 8 MAC-T IVV-08_151
Achievements to Date • Formal assertion notation identified u • • u Most work with LTL Extended to work with Live Sequence Charts (LSC) Objective validation metrics defined u Requirements, Antecedent, Unique First Cause, and Unique Cause Test case generation tool developed u Developed tool generating tests from LTL n • • • u Capable of generating tests to all metrics defined Prototype tool working on LSC developed Tool Demo in Tech. Session Developed test-adequacy measurement tool for the defined validation metrics Evaluation of metrics and tool 12 papers and one Ph. D dissertation (Ajitha Rajan) SAS_08_Model_Val_Exec_Heimdahl 9 MAC-T IVV-08_151
Next Steps • Investigate alternative requirements • notations to LTL Complete empirical evaluation of the effectiveness in model validation Flight Guidance System (FGS) evaluation complete u Display Manager (DM) evaluation in work u Coordinate evaluation on NASA IV&V project u • Coordinate technology transfer SAS_08_Model_Val_Exec_Heimdahl 10 MAC-T IVV-08_151
- Slides: 10