Modelbased Design Verification IVV Facility IVV Annual Workshop

Model-based Design Verification IV&V Facility IVV Annual Workshop September, 2009 Tom Hempler

Agenda • • • 3/5/2021 IV&V Facility Design & Integration Verification Planning Iterative Verification Process Perspectives on Verification As-Built Design Verification Goals and Objectives Design Verification Process Design Verification Methods 2

Software Design & Integration Verification Work Plan IV&V Facility • Using criticality and risk analysis results from PBRA, plan IV&V verification tasks • Review risk assessment results from PBRA – Review software, interface, and integration design risks identified by the PBRA – Review and target developer artifacts related to mission critical and safety critical behavior – Identify critical software design and integration topologies and protocols – Review schedule risks associated with development schedule and milestones – Integrate design verification schedule with engineering services master schedule 3/5/2021 3

Iterative Verification Process IV&V Facility • Design verification is accomplished iteratively at multiple levels during all lifecycle phases • Evaluate how well software components interact and support validated capabilities, limitations, and performance requirements. • The System Reference Model (SRM) contains the structure and behavior used as a baseline for verification – Mission and safety critical software components and their interactions are compared with as-designed software components – SRM component operations, attributes, and services are functionality and performance requirements of the system – Non-functional requirements such as reliability, safety, scalability, recoverability, security, and availability • Perspectives on the design along with scenarios facilitate verification and reveal program/technology risks 3/5/2021 4

IV&V Facility System Under Development Functional Capabilities & Limitations Structure Behavior Relationships Services System Reference Model Physical Constraints Software and Interface Design Verification Views DESIGN VERIFICATION - RELATIONSHIPS 3/5/2021 5

Relationships Perspective IV&V Facility • Relationships defined as having size, weight, physical features, and performance properties • Relationships among capabilities and limitations are dependencies, associations, and design or performance constraints. • Relationships among physical components are interactions, services provided/consumed, containership, ownership, controller-terminal roles, etc. • Verification is accomplished among all critical relationships described in SRM structure and behavior • Diagrams: – Structure Diagrams – Behavior Diagrams 3/5/2021 6

IV&V Facility System Under Development Functional Capabilities & Limitations Structure Behavior Relationships Services System Reference Model Physical Constraints Software and Interface Design Verification Views DESIGN VERIFICATION - PHYSICAL 3/5/2021 7

Physical Perspective IV&V Facility • Features: – Describes the physical features and interrelationships among software components – Useful for identifying software and hardware; and software component interdependencies – Describes deployment configurations in terms of multiple instantiations, redundancy, and their interaction – Describes systems, sub-systems and components using classes; and their capabilities and limitations using attributes – Logical design provides facility to describe software domain; and their capabilities and limitations using attributes • Diagrams: – Structure Diagrams with design and performance constraints – Behavior Diagrams 3/5/2021 8

IV&V Facility System Under Development Functional Capabilities & Limitations Relationships Services System Reference Model Structure Behavior Physical Software and Interface Design Verification Views DESIGN VERIFICATION - CAPABILITIES & LIMITATIONS 3/5/2021 9

Capabilities & Limitations Perspective IV&V Facility • Features: – SRM structure provides facility to describe software domain; and their capabilities and limitations using attributes – Accounts for non-functional properties such as availability, reliability (fault tolerance), availability, safety, performance (throughput), and scalability – Addresses resource sharing, concurrence – Useful for identifying design where processes can be controlled (initialization, recovery, reconfiguration, shut down) – Provides facility to verify performance, safety, dependability and other non-functional characteristics of systems, sub-systems and components • Diagrams: – Structure Diagrams with design and performance constraints 3/5/2021 10

IV&V Facility System Under Development Functional Capabilities & Limitations Relationships Services System Reference Model Structure Behavior Physical Software and Interface Design Verification Views DESIGN VERIFICATION - FUNCTIONAL 3/5/2021 11

Functional Perspective IV&V Facility • Features: – Systems and sub-system components (e. g. , program libraries, or architecture layers) are allocated operations that define the functionality of the design – Provides a look into the environment that surrounds the system and the effects (adverse or otherwise) it has on it – Serves as a basis for viewing validated requirement allocation and human interface allocations to the system – Useful for identifying work done by teams/individuals, cost evaluation and planning, progress monitoring, reasoning about software reuse, portability and security – Domain model contains operations or activities that describe systems, sub-systems and components using classes, and their capabilities using attributes and operations • Diagrams: – Structure diagrams – Behavior diagrams 3/5/2021 12

IV&V Facility System Under Development Functional Capabilities & Limitations Relationships Services System Reference Model Structure Behavior Physical Software and Interface Design Verification Views DESIGN VERIFICATION - SERVICES 3/5/2021 13

Services Perspective • Features: – Uses a set of use cases, activities or sequences of interactions between objects and between processes that weaves a behavioral thread through the design – Useful for realizing architectural elements used during software and interface design – Also used for verification of interface design to identify preventative measures taken for adverse behaviors • Diagrams: – Structure Diagrams – Behavior Diagrams 3/5/2021 14 IV&V Facility

Design Verification Goal/Objectives IV&V Facility • Goal – Ensure that the proposed software and integration design adequately satisfies the software architecture and validated requirements and behavior. An adequate design is determined by assessing it for completeness, correctness, consistency, ambiguity, and testability. • Objectives – Ensure the integration design is a correct, accurate and complete transformation of the validated behavior – Ensure that no unintended features were introduced during the design phase and that the design guards against undesirable behavior – Ensure that the Integration design is consistent with the verified software architecture and validated behavior – Ensure that the proposed design is testable – Verify interfaces define services to be provided and/or consumed, the preconditions for invoking the interface, the post conditions, and the invariants. – Ensure design documentation is acceptable and will support follow-on verification and future maintenance activities – Verify that the interface design isolates behavioral components from each other, and from computational components and data stores. 3/5/2021 15

Answering Three Questions IV&V Facility • Using validated requirements and behavior – Verify that the system software design provides for the specified capabilities, limitations, and behavior. – Verify that the system software design guards against undesirable behavior – Verify that the system software design provides self protection and recovery capability against adverse conditions 3/5/2021 16

IV&V Facility DESIGN VERIFICATION PROCESS 3/5/2021 17

Definitions for Software Design and Integration Quality Factors IV&V Facility Design verification check list • Unambiguous – – • Correct – – • – – – States and state transitions are used similarly throughout the design. For example, if event ε triggers Object 1 to transition to state A, then elsewhere in the design that same event shall not cause Object 1 to transition to a state other than A. All terms and concepts are used similarly within the design itself, as well as with external artifacts such as the system and software architectures. Testability – 3/5/2021 There is a logical decomposition into subsystems and modules, and their interactions are specified. The hardware, software, and user interfaces are specified to an appropriate level. An appropriate level is one that identifies the information being processed, communication mechanisms and protocols, and services that particular subsystems or modules provide (i. e. , behaviors that are changed or affected based on communication between two modules). Functionality (e. g. , algorithms, state/mode definitions, input/output validation, exception handling, reporting, and logging) is specified at the appropriate level of decomposition. Performance criteria (e. g. , timing, sizing, speed, capacity, accuracy, precision, safety, and security) are specified at the appropriate level of decomposition. Consistent – • The software design satisfies the software architecture and validated requirements (i. e. , behaviors in the SRM, as well as non-functional requirements like safety or other “…ility”-like requirements). The software design complies with applicable Mission and NASA standards, references, and policies. Complete – – • The documentation is legible, understandable, and could result in only one interpretation by the intended audience. All acronyms, mnemonics, abbreviations, terms, symbols, and special design languages are defined. There are objective acceptance criteria such that the design can be shown to pass or fail. 18

Design Verification Process • • IV&V Facility Constraints Task Preconditions – Software architecture verification has been started – SRM products necessary to support design verification have been completed and validated – A criticality assessment has been made • PBRA determined the prioritization and scope of the verification effort, including the level of analysis applied to each component/behavior – Artifacts necessary to support software design verification provided by Project • Task Inputs – Software design material – Validated software requirements – Verified software architecture – SRM elaborated to a level indicative of software detailed design • Task Outputs/Deliverables – IV&V evaluation of project’s responses to identified deficiencies/issues are documented & tracked – task status, schedules & risks documented & reported on regular basis – Report that documents IV&V assessment of software design – Model Correlation Matrix has been updated based on verified software design – Findings/recommendations for SRM changes/additions 3/5/2021 19

Design Verification Methods • • 3/5/2021 IV&V Facility Inspection Analysis Demonstration Test 20

Design Verification - Inspection IV&V Facility • Inspection – Review and compare artifacts with validated behavior – Verify that the design can be implemented – Verify that the design is traceable to the validated requirements and behavior – Verify that software design and integration design is complete and correct – Verify that omissions, defects, and ambiguities in the design are detected and recorded. 3/5/2021 21

Design Verification - Analysis IV&V Facility • Analysis – Verify through software structure analysis that the design integrity goals are met and interrelationships described in the behavior model exist – Verify that fault recovery strategies described in the validated behavior model are employed. – Verify that the design is traceable to the validated requirements and behavior – Verify that software design and integration design is complete and correct – Verify that omissions, defects, and ambiguities in the design are detected and recorded. 3/5/2021 22

Design Verification - Demonstration IV&V Facility • Demonstration Evaluation – Verify that software and integration design is testable within the behavior model – Verify relationships between the validated behavior and the software component features and integration properties – Apply assertions used on the logical model to components of the physical model – Verify that the design is traceable to the validated requirements and behavior – Verify that the design supports critical capabilities – Verify that integration logic is complete and correct – Verify that omissions, defects, and ambiguities in the design are detected and recorded. 3/5/2021 23

Design Verification - Test IV&V Facility • Test and Evaluation – Verify that software and integration design is testable within the behavior model – Verify relationships between the validated behavior and the software component features and integration properties – Apply assertions used on the logical model to components of the physical model – Verify that the design is traceable to the validated requirements and behavior – Verify that the design supports critical capabilities – Verify that integration logic is complete and correct – Verify that omissions, defects, and ambiguities in the design are detected and recorded. 3/5/2021 24

Summary IV&V Facility • Design & integration verification planning is a critical first step in an executable process • Iterative verification processes are necessary to accomplish verification at all levels of design • Perspectives on verification are necessary to maintain context and accomplish goals • Verification goals and objectives are defined • Design verification process is implemented • Design verification methods are modelbased, independent , and objective 3/5/2021 25

IV&V Facility QUESTIONS? 3/5/2021 26