Verification and Validation for Systems Software and Hardware

Verification and Validation for Systems, Software and Hardware V&V Edward A. Addy, Ph. D, PMP

010001 10100001 11010001101 2

Acknowledgement The work described in this presentation is based on IEEE Standard 1012, which is developed by the IEEE P 1012 Working Group that is chartered by the Software & Systems Engineering Standards Committee (C/S 2 ESC) of the IEEE Computer Society. The Working Group has voluntary participation by members of the systems and software engineering communities. While based on IEEE 1012, the contents of this tutorial have been developed by the author and do not necessarily represent the views of the IEEE Computer Society or the IEEE Standards Association. 3

Topics q q q q q 4 Background of V&V System Life Cycle Processes V&V Tasks V&V, QA, Testing Application of the Standard V&V and Systems Engineering System-of-Interest System Life Cycle – Recursive Application System of Systems Summary

Origins of Verification and Validation Software V&V has its roots in Systems Engineering principles from the early days of satellite development in the 1960’s and 1970’s IEEE Standard 1012 was one of the first IEEE standards published in the areas of systems and software engineering, with the original 1986 version focused on the Software V&V Plan Versions in 1998 and 2004 changed the focus from the Software V&V Plan to Software V&V processes The version published in 2012 increased the scope to also include system and hardware V&V A new version was published in October 2017 (IEEE STD 10122016 with Cor 1 -2017); this version aligns with the process framework of ISO/IEC/IEEE 15288: 2015 and ISO/IEC 12207: 2008 5

Definition: Verification and Validation Processes Verification process: provides objective evidence for whether the products successfully complete each life cycle activity and satisfy all the criteria for initiating succeeding life cycle activities Validation process: provides objective evidence for whether the products satisfy intended use and user needs Verification – The product is built correctly Validation – The correct product is built The verification and validation (V&V) processes • Interrelated • Complementary • Have activities performed throughout the life cycle of the system 6

Who Performs V&V? • V&V activities and tasks, including analysis, evaluations, and tests, may be performed by multiple organizations (e. g. , development, project management, quality assurance, and V&V) • Parameters of Independence • Technical • Managerial • Financial • Forms of Independence • Embedded – V&V conducted within development org • Internal – V&V conducted by development org working separately, benefiting from staff with knowledge of the system • Integrated – Technical independence, but working with development org for rapid feedback • Modified – Prime integrator manages V&V effort; compromises managerial independence • Classical IV&V – Separate V&V effort managed by org outside the development org 7

System Life Cycle Technical Processes Derived from ISO/IEC/IEEE 15288: 2015 8

System Life Cycle Processes vs. Life Cycle Models • Each system has a life cycle defined by its life cycle model and broken into stages. Each stage is a major life cycle period based on progress and achievement milestones of the system. • Typical system life cycle stages include concept, development, production, utilization, support and retirement. • System life cycle processes can be arranged by the organization responsible for the system to describe the system life cycle model, such as: Waterfall Spiral Iterative • The system progresses through the stages of its life cycle as a result of actions using the life cycle processes. 9

Concepts Life Cycle Process V-Model (Stages) User s Need Ops & Maint Accept Test Sys Reqs Qual Test esign Arch Integr Test D Unit Test Implementation 10 Retire The Implementation process results in a realized system element, i. e. , a system or a software or hardware element. Verification Validation

System V&V Task Overview System Life Cycle Processes Implementation Integration Transition Operations Maintenance Disposal • Business or Mission Analysis Results Evaluation • Stakeholder Needs and Requirements Evaluation • Interface Analysis • Architecture Evaluation • Interface Analysis • Requirements Allocation Analysis • Design Evaluation • Interface Analysis • Implementation Strategy Assessment • System Element Implementation Analysis • System Element Interaction Analysis • System Integration Strategy Assessment • Transition Strategy Evaluation • Transition Demonstration Assessment • Operating Procedure Evaluation • System Maintenance Strategy Assess • System Maintenance Execution Assessment • Disposal Plan Evaluation Conformance & Scoping Traceability & Criticality Analyses Traceability & Criticality Analyses Criticality Analysis Test System Analysis Strategy and Results Evaluation Design Definition Product Evaluation & Assessment Test Planning an d Execution Business or Mission Analysis Stakeholder Needs and Rqmnts Def System Requirements Def Architecture Definition System V&V Tasks (Minimum Requirements) System & Environment Issues Hazard, Security & Risk Analyses Hazard, Security & Risk Analyses Finding problems as early as possible decreases rework and the cost to fix problems 11

Conformance and Scoping V&V Tasks Traceability Analysis • Each requirement is accomplished by one or more elements at the next lowest level • The relationship is maintained between the requirement and the elements that satisfy it (bi-directional) Criticality Analysis • The importance of the system and each of its elements to the user and the acquirer is determined, based on complexity, criticality, risk, safety level, security level, desired performance, reliability, or other system unique characteristics • Integrity levels are used to determine the V&V tasks, activities, rigor, and the level of intensity of the V&V to be performed 12

Criticality Analysis Determines Integrity Levels that are Mapped to V&V Tasks • Conformance with IEEE 1012 requires Criticality Analysis to determine the Integrity Level of the system and each element – Integrity Level is based on the importance of the system to the user and acquirer – Criticality Analysis considers various aspects such as complexity, risk, safety level, security level, desired performance, reliability, or other project-unique characteristics • 1012 -2012 uses Integrity Levels to determine the minimum V&V tasks to be performed – V&V tasks are assigned based on Integrity Level, with more tasks and activities being conducted at higher Integrity levels – Integrity Levels are also used to determine the rigor and level of intensity for V&V tasks and activities • The assigned Integrity Levels may change as the system evolves, so Criticality Analysis is repeated during development Look in the most important places, and make sure you know when there is a change in what is important. 13

Integrity Levels Integrity Level High V&V Tasks: Larger number of V&V tasks and more comprehensive Rigor: More formalism and disciplined approach Intensity: Larger effort and use of modeling/simulations and automated tools V&V Tasks: Smaller number of V&V task, mostly reviews Rigor: Evaluations involve just review of development artifacts or participation in walk-throughs Low 14 Intensity: Mostly manual efforts

System and Environment Issues V&V Tasks Hazard Analysis • Identify potential solution hazards and assess the consequences of each hazard • Update the analysis at each level as more information is available Security Analysis • Review security solutions to determine acceptable levels of risk • Update the analysis at each level as more information is available Risk Analysis • Identify technical and management risks • Determine mitigation to reduce risk to acceptable level • Perform mitigation tasks and monitor risks 15

Environment and Other Considerations Operational • Normal (temp, EMI, pressure, vibration, etc. ) • Disaster considerations • Error Detection & Recovery Industry/Government Policies & Regulations System Quality/Manufacturing Attributes & Tolerances Other Considerations • Legal • Operator Training • Technology 16 System Risks • Hazards • Security • Technical

Example Software V&V Tasks & Activities From Table 1 c— V&V tasks, inputs, and outputs 9. 1 Activity: Software Concept V&V (Software, 12207 - Software Requirements Analysis process) V&V Tasks (1) Concept Documentation Evaluation V&V subtasks Required Inputs Concept documentation System architectural design Supplier development plans and schedules Required Outputs Task report(s) – Concept documentation evaluation Anomaly report(s) User needs Acquisition needs (4) Criticality Analysis V&V subtasks (5) Hazard Analysis V&V subtasks 17 Concept documentation (system requirements) Task report(s) – Criticality analysis Developer integrity level assignments Anomaly report(s) Concept documentation Task report(s) – Hazard analysis Anomaly report(s)

Example Minimum V&V Tasks Assigned to Integrity Levels From Table 2 c— Minimum V&V tasks assigned to each integrity level for software V&V Activities Activity: Software Concept V&V (see 9. 1) Activity: Software Requirements V&V (see 9. 2) Integrity Levels 3 X 2 X 1 4 3 2 1 Concept Documentation Evaluation 4 X Criticality Analysis X X X X Design Evaluation of New Constraints Hazard Analysis Installation Checkout Installation Configuration Audit Interface Analysis 18 X

System Requirements Definition V&V 19 System Requirements Performance Requirements • Function #1 • Function #2 • Function #n Interface Requirements • Internal Requirements • External Requirements Non-Functional Requirements • Quality • Safety • Security • Supportability • Size Constraints/Intended Environment • Normal • Error Conditions • Disaster Conditions • Stress Conditions System Requirements Evaluation V&V Test Planning Traceability Analysis Criticality Analysis Risk Analysis Security Analysis Hazard Analysis Interface Analysis Scenarios for V&V Analysis

Control Flow Diagram: Shuttle Open/Close Function: Control Open/Close Doors 1. When the shuttle stops, the shuttle shall issue a command to open the doors. 2. The shuttle shall maintain the doors open for 30 seconds. 3. After waiting 30 seconds, the shuttle shall close the doors and issue a command to start shuttle motion. Sense Stop Issue Command to Open Door 30 m Secs Hold Door Open Issue Command to Close Door and Start Shuttle Motion 20

Control Flow Diagram: Analysis Revision Sense Stop Sense Terminal Location Issue Command to Open Door Sense Door Status Door Open ? Y Hold Door Open N Maintain Shuttle in Safe Mode & Notify Operator 30 m Secs Sense Door Blockage Door Free? N Maintain Shuttle in Safe Mode & Notify Operator Y Sense Door Status Issue Command to Close Door Issue Command to Start Shuttle Motion 21 Y Door Closed ? N Maintain Shuttle in Safe Mode & Notify Operator

Architecture & Design: V&V of the Curiosity Rover Entry, Descent, and Landing System Curiosity was five times the weight of previous Mars Exploration Rovers, had a landing ellipse 15% of the area, and was to land at a higher elevation. This required a new architecture. • Airbags could not support the weight • Propulsion system could not be mounted under the lander • The Sky Crane design was developed to separate the Rover from the descent stage attached by a bridle Not feasible to replicate environmental conditions of Mars atmosphere and terrain • Heavy dependence on simulations • Limited use of testing using the actual Curiosity spacecraft, where exercising the actual flight hardware was crucial to the test • V&V decomposed into three distinct domains: flight dynamics, flight system, and subsystems From “Verification and Validation of the Mars Science Laboratory/Curiosity Rover Entry, Descent, and Landing System” at https: //arc. aiaa. org/doi/full/10. 2514/1. A 32680? mobile. Ui=0& 22

Architecture & Design Issue: Feasibility 23

Software Construction V&V Source Code, Software Design, Interface Design, Coding Standards Source Code Units • Unit #1 • Unit #2 • Unit #n Software Design • Design Decisions • Design Constraints • Design Assumptions Interface Design • Internal Interfaces • External Interfaces Constraints/Intended Environment • Normal • Error Conditions • Disaster Conditions • Stress Conditions 24 Source Code and Source Code Documentation Evaluation V&V Test Cases, Test Procedures and Component Test Execution Traceability Analysis Criticality Analysis Risk Analysis Security Analysis Hazard Analysis Interface Analysis Scenarios for V&V Analysis

Software Construction: Traceability Analysis Stakeholder Requirements System Requirements Stk Req #1 Sys Req #1 Stk Req #2 Sys Req #2 Design Implementation Tests Arch #1 Des #1 Imp #1 Arch #2 Des #2 Test #1 Test #2 Test #3 Test #4 Architecture Arch #3 Des #4 Des #5 Sys. Req #j Arch #k 25 Imp #3 Imp #4 Arch #4 Stk Req #i Imp #2 Des #m Imp #5 Imp #6 Test #5 Test #6 Test #7 Imp #7 Test #8 Imp #n Test #p

Software Construction: Discrete Mathematics Position from Earth NASA long range probes were beginning to drift off course several years into flight, slightly at first but with increasing divergence Simulators were able to duplicate these effects using a copy of the flight software Navigation software determined delta changes for distance from Earth and added the deltas to cumulative values. Eventually the deltas were comparatively small so that the added deltas values were truncated. The solution was to save several delta values until their sum could be added without truncation. 26 Time from Launch

Software Construction: Interface Units of Measure There are multiple examples of system failures caused by interfaces where data is transmitted with the value in one unit but received with the interpretation of the value in a different unit Mars Climate Orbiter English measurements in pounds were not converted to metric measurements in Newtons for thruster data, resulting in the Orbiter flying off course and disintegrating. Tokyo Disneyland's Space Mountain Master plans were converted from English units to metrics units in 1995, but in 2002 replacement axels were ordered using the pre-1995 values; the axel broke in mid-ride. 27

What V&V is needed for Reuse? Reuse includes a wide range of scenarios, including commercial products, previously developed products (from other projects, from partners, from customers), and product line assets One of the tasks in Acquisition Support V&V is to determine the extent of V&V on reuse items selected for the program. V&V may have occurred during domain engineering and the artifacts from this V&V effort may be included. The real issue is knowing what has changed in the reused asset, what has changed in the system in which the reused asset is being placed, and what has changed in the environment. 28

Therac 25 Error: Timing/Data Incoherency Therac 6 and 20 6 Me. V or 20 Me. V radiation Hardware Safety Error Interlocks Simple patient data computer interface (character transfer) Radiation treatment implemented by hardware • • • 29 Therac 25 10 Me. V or 25 Me. V radiation Reuses patient data entry software Removed Hardware Safety Interlocks No explanation of error codes in Users’ Manual 6 patients overdosed (4 deaths) Patient data entry software reused was not compatible with Therac 25 design Operator use of patient data entry edit feature and rapid data entry creates scenario for problem Edit feature resets radiation level to max (25 Me. V treatment) Rapid typing and data entry exceed character transfer computation Error code “Malfunction 54” not explained in manual (radiation dosage undetermined)

Ariane 5 Error: Data Overflow • The Ariane 5 was the successor to the Ariane 4 rocket, but used much of the same software as in the Ariane 4. • Ariane 5 disintegrated 39 seconds into its maiden flight. • Analysis indicated the guidance computer attempted to convert the sideways velocity of the rocket from a 64 -bit floating point format to 16 -bit signed integer format; however, the velocity was larger than the maximum value of 32, 767 that could be stored in signed integer format. • The error caused the nozzles of two solid rocket boosters and an engine to suddenly swung out of position, triggering a self-destruct mechanism. • Most of the conversion software contained data checking and recovery protection, but this conversion lacked protection because it had been deemed safe since the Ariane 4 had never had a problem. 30

Is there any difference between V&V and QA? • ISO/IEC/IEEE 15288 includes the Quality Assurance Process as one of the Technical Management processes, while Verification and Validation are included in the Technical processes. • IEEE 730 Software Quality Assurance Processes says that QA is intended “to produce and collect evidence that form the basis for giving a justified statement of confidence that the software product conforms to its established requirements. ” • Verification provides objective evidence for whether the products satisfy all the criteria for initiating succeeding life cycle activities; Validation provides objective evidence for whether the products satisfy intended use and user needs. The focus of QA is conformance, while the focus of V&V is correctness. 31

Relationship between V&V and QA • QA is often a part of the development organization; V&V is usually separate from the development organization, depending on the level of independence. • V&V and QA may use some of the same methods, such as reviews of system artifacts and analysis of test results. • V&V tasks may be performed by the QA organization, depending on the level of independence required. • V&V may review analysis and other outputs of the QA process as part of V&V tasks. • QA may assess the V&V process as part of its scope, but it is not within the normal scope of V&V to review the QA process. 32

Isn’t V&V just Testing? • Testing is certainly one of the techniques that is used by V&V. • Activities that are generally considered to be testing can be performed only upon reaching implementation of one or more elements. • In order to discover errors as early as possible, V&V activities occur early in the life cycle of the system • During Concept Stage • • Business or Mission Analysis Results Evaluation Hazard Analysis Security Analysis Risk Analysis • During Requirements Stage • Stakeholder Needs and Requirements Evaluation • Interface Analysis 33

Relationship between V&V and Testing? • Testing is always a responsibility of the development organization. • Testing is inherently a verification or validation activity. • For a system of low integrity level, testing may be the only necessary V&V activity. • For systems of high integrity level, V&V should be responsible for testing that is independent of the development organization. 34

Testing Example (a very simple example) A software function must ensure that an input string consists of seven alphanumeric characters. What are the necessary test cases for this function? • String of 7 alphanumeric characters • • String of 7 lower case characters String of 7 upper case characters String of 7 numerals String of mixed lower case characters, upper case characters and numerals • String of alphanumeric characters whose length is less than 7 • String of zero characters • String of alphanumeric characters whose length is greater than 7 • What should be the maximum length tested? • String of seven characters with at least one character that is not alphanumeric • • 35 String with a special character (e. g. , !@#$%^&*-_+) String with a space String with a symbol (e. g. , ) Should this case also be repeated with strings of length less than and greater than 7?

Hubble Telescope • • 36 The Hubble telescope deployed in 1990 was found to produce images that were much more blurry than expected. Tests by NASA showed the Hubble primary lens had a spherical aberration, meaning it was slightly the wrong shape. (The error has been subsequently corrected by installing a series of small mirrors to correct the flaw. ) The developer tested the Hubble lens using a test instrument called the ‘reflective null corrector’, which was determined to be about a millimeter askew. This is a very large error for lens testing equipment, with which positions are often measured to a fraction of the wavelength of light (less than a thousandth of a millimeter). Lens testing is usually conducted using two or three test instruments, and testing continues until all instruments are in agreement. The developer did use another test instrument, a smaller refractive null corrector that used lenses rather than mirrors. The refractive instrument was much cruder, and was neither certified to measure spherical aberration, nor intended for precise measurement of the final surface figure. The differing results from the refractive null corrector were ignored, but were eventually proven correct.

System, Software and Hardware Testing Software 4 3 2 1 V&V Software Component Testing Perform Review No action V&V Software Integration Testing Perform Review No action V&V Software Qualification Testing Perform Review No action V&V Software Acceptance Testing Perform Review No action Hardware V&V Testing by Integrity Level 4 3 2 1 V&V Hardware Component Testing Review No action V&V Hardware Integration Testing Review No action V&V Hardware Qualification Testing Perform Review No action V&V Hardware Acceptance Testing Perform Review No action System 37 V&V Testing by Integrity Level 4 3 2 1 V&V System Integration Testing Perform Review No action V&V System Qualification Testing Perform Review No action V&V System Acceptance Testing Perform Review No action

Application of IEEE 1012 Common V&V Processes (Clause 7) System V&V Processes (Clause 8) System V&V Software V&V Processes (Clause 9) Hardware V&V Processes (Clause 10) System, Software, & Hardware V&V 38

What is the difference between V&V and Systems Engineering? • V&V is a subset of Systems Engineering • Systems Engineering produces the artifacts of the system • Its general focus is on the system performing its intended functionality • V&V provides an objective assessment on the artifacts of the system • Its greatest value is often from considering boundary conditions, stress conditions, anomaly inputs, and multiple condition faults • There is an analogy with building a home • The contractor is responsible for constructing the home according to the plans (developer) • The county inspector reviews the home being constructed to ensure it meets applicable codes (quality assurance) • The home owner may hire an inspector to assess the home being constructed to ensure it meets the plans and is providing the homeowner what they expected. 39

System of Interest The Implementation process results in a realized system element, i. e. , a system or a software or hardware element. Each element undergoes its own full life cycle. Recursive, iterative application of life cycles System-of-Interest Implementation Process System V&V 40 System Implementation Process System V&V Software element Software V&V Software element Hardware element Software V&V Hardware V&V

System Development Occurs using a Recursive Application of Life Cycle Processes The System Implementation Process results in a realized system element, i. e. , a lower-level system or a software or hardware element. System-of-Interest Implementation Process System V&V 41 System Implementation Process System V&V Life cycles are recursive and iterative. V&V addresses the collective results of: • Environmental Requirements • Operational Issues • Legal and Policy Constraints • Security and Safety • Training and Human Interface Software element Software V&V Software element Hardware element Software V&V Hardware V&V

System Life Cycle with Embedded Software and Hardware Life Cycles 42 The purpose of the system Implementation process is to realize a specified system element (system, software or hardware) Within the system Implementation process, each element is developed using a full system, software or hardware life cycle

Embedded, Recursive Life Cycles in the System Life Cycle Stakeholder Needs Maintenance & Disposal Sys Reqm’ts Operations Architecture Design Element Life Cycles Acceptance Implementation & Integration Element 43 Qualification Element

Challenges for V&V of a System of Systems 1. System elements operate independently. – Each system in an System of Systems (So. S) is likely to be operational in its own right 2. System elements have different life cycles. – An So. S involves more than one system element. Some of the system elements are possibly in their development life cycle while others are already deployed as operational. In extreme cases, older systems elements in an So. S might be scheduled for disposal before newer system elements are deployed. 3. The initial requirements are likely to be ambiguous. – The requirements for an So. S can be very explicit for deployed system elements. But for system elements that are still in the Development Stage, the requirements are usually no more explicit than the system element requirements. Requirements for an So. S mature as the system elements mature. From the INCOSE Systems Engineering Handbook v. 3. 2. 2 44

Key Benefits of V&V • IEEE Std 1012 provides criteria for conformance yet permits flexibility in implementation via o o o • • • Early anomaly detection o Facilitates early detection and correction of anomalies Management insight into development status o Enhances management insight into process and product risks Compliance to schedule, technical performance requirements, and budget o 45 Integrity level application Intensity and rigor application Degree of conformance established by regulatory or acquiring organization Supports the life cycle processes to ensure conformance to program performance, schedule, and budget

Verification and Validation IEEE 1012 Verification and Validation of Systems, Software and Hardware ? The author may be contacted at edward. addy@ngc. com 46

About the Presenter Dr. Addy has been the Chair of the P 1012 Working Group since 2018; this group is sponsored by the IEEE Computer Society Software and Systems Engineering Standards Committee (S 2 ESC) and is responsible for revisions to IEEE Standard 1012 Verification and Validation for Systems, Software and Hardware. He served as Working Group Secretary and Editor from 2007 through 2017, and has been a member since 2005. Dr. Addy is a co-author of Reuse-based Software Engineering: Techniques, Organization, and Measurement (John Wiley & Sons, 1999), and of papers in the areas of software assurance, software product lines, and software safety. He is an elected member of the S 2 ESC, a Computer Society Representative to the IEEE Systems Council, and Chair of the Computer Society Richard E. Merwin Scholarship Committee. He has a Ph. D in Computer Science from West Virginia University, and holds PMP certification. He is a Senior Systems Engineer with Northrop Grumman Mission Systems. 47