Planning for and Managing Software Verification Validation VV

Planning for and Managing Software Verification & Validation (V&V) Quality Assurance Project Oversight Jeff Lewis, PMP The Æon Group, Inc. Jeff. Lewis@The. Aeon. Group. net jlewis@ieee. org

References • Materials used in developing this presentation – Institute of Electrical and Electronics Engineers (IEEE) • www. ieee. org • IEEE Standard 1012 -1998, IEEE Standard for Software Verification and Validation • IEEE Standard 1058 -1998, IEEE Standard for Software Project Management Plans (this Standard has been updated by ISO/IEC/IEEE 16326 -2009) • IEEE Standard 12207, Standard for Information Technology – Software Life Cycle Processes (this Standard has been updated by ISO/IEC/IEEE 15288 -2008) – State of California • www. ocio. ca. gov • California Project Management Methodology (CA-PMM) • California Project Oversight Framework – Carnegie Mellon University Software Engineering Institute (SEI) • www. sei. cmu. edu • Software Capability Maturity Model (CMM) – Project Management Institute • www. pmi. org • A Guide to the Project Management Body of Knowledge Planning for and Managing VV QA POC J. Lewis 2

Agenda • Background & Definitions – – – Software Verification & Validation Quality Assurance Oversight Level (Degree) of Independence Integrity Levels • Comparison of V&V, Oversight, & QA – Focus • Planning for & Managing V&V, Oversight, & QA – Defining Tasks (Scope) for V&V, Oversight, & QA – Considerations for Managing V&V, Oversight, & QA • Summary Planning for and Managing VV QA POC J. Lewis 3

Background (Foundations) Planning for and Managing VV QA POC J. Lewis 4

Project Management Plan Processes • Management Methods & Processes • Supporting Plans & Processes – – – – Startup, Work, Control, Closeout – Change – Risk – Reviews – Reporting – Communications – Human Resources • Technical Methods & Processes – Methods for producing product Planning for and Managing VV QA POC J. Lewis 5 Scope Requirements Schedule Cost Verification and validation Configuration management Quality assurance Software documentation Reviews and audits Problem resolution Subcontractor management Process improvement Procurement

Software V&V Planning for and Managing VV QA POC J. Lewis 6

Software Verification & Validation IEEE Standard 1012 -1998 • Software Verification process provides objective evidence whether the software and its associated products and processes – Conform to requirements for all life cycle activities during each life cycle process – Satisfy standards, practices, and conventions during life cycle processes – Successfully complete each life cycle activity and satisfy all the criteria for initiating succeeding life cycle activities • Software Validation process provides evidence whether the software and its associated products and processes – Satisfy system requirements allocated to software at the end of each life cycle activity – Solve the right problem – Satisfy intended use and user needs Planning for and Managing VV QA POC J. Lewis 7

V&V Tasks by Life Cycle Phase IEEE Standard 1012 -1998 Planning for and Managing VV QA POC J. Lewis 8

Optional V&V Tasks IEEE Standard 1012 -1998 • Regression Analysis and Testing • Reusability Assessment Ø Security Assessment • Simulation Analysis Ø Sizing and Timing Analysis • System Software Assessment • Test Certification • Test Evaluation Ø Test Witnessing • Training Document Evaluation • User Documentation Evaluation • User Training • V&V tool plan generation Ø Walkthroughs – Design – Requirements – Source code – Test • Algorithm Analysis • Audit Performance • Audit Support • Control Flow Analysis Ø Cost Analysis • Database Analysis • Data Flow Analysis • Disaster Recovery Plan Assessment Ø Distributed Architecture Assessment Ø Feasibility Study Evaluation • Independent Risk Assessment • Inspection • Operational Evaluation • Performance Monitoring • Post Installation Validation Ø Project Management Oversight Support • Qualification Testing Planning for and Managing VV QA POC J. Lewis 9

Quality Assurance Planning for and Managing VV QA POC J. Lewis 10

Quality Assurance • Multiple Definitions & Contexts – Quality Management – Quality Control – Quality Assurance • Quality – Consistently meeting or exceeding customer expectations – The degree to which a set of inherent characteristics fulfill requirements Planning for and Managing VV QA POC J. Lewis 11

Software Quality Assurance • Provides management with appropriate visibility into the process being used and the products being built • Verifies compliance with project requirements • Involves – Reviewing or auditing products and processes – Providing appropriate visibility to affected groups and Senior Management – Identifying, documenting, and tracking noncompliance issues • Based on a documented plan and processes Planning for and Managing VV QA POC J. Lewis 12

SQA Functions • Ensure SQA activities exist for all software development and maintenance projects • Objectively and independently verify that all software products and activities adhere to the applicable standards, procedures, and requirements • Inform all projects of SQA activities and results • Seeks to resolve non-compliance issues with the software project – For issues not resolvable within the software project, the SQA function escalates the issue to an appropriate level of management • Acts in an advisory role in clarifying and understanding applicable software procedures and practices Planning for and Managing VV QA POC J. Lewis 13

SQA Methods & Techniques • Audit – Process Compliance • Project Management • Supporting Roles (CM, IV&V, Contracts, Documentation) – Checklist • Product Development • Peer Reviews • Collect Statistics – Compliance – Rework – Process Effectiveness Planning for and Managing VV QA POC J. Lewis 14

QA for a Software Project 1. Develop Plans – Project Management Plan • Development Processes • Management Processes • Supporting Processes – Risk Management – Configuration Management – Quality Assurance Plan 2. Follow Plans – Process Audits – Compliance Checklists 3. Measure Success – Metrics Planning for and Managing VV QA POC J. Lewis 15

Process Compliance Checklist (Example) • Project Management Plan Process Compliance – Has the Project Management Plan (PMP) been approved by the ITEC and customer? – Does the PMP follow ANSI/IEEE 1058 -1998? – Does the PMP clearly state the project deliverables? – Does the PMP clearly identify project start and finish dates? – Does the PMP contain a project charter and project objective statement? – Is the approved PMP entered in the CM library? –. . . Planning for and Managing VV QA POC J. Lewis 16

Architecture Design Review Checklist (Example) 1. 1 Context Diagram 1. 1. 1 Has every external device, software system, or subsystem with which your system must communicate been represented as a terminator? List those which have not. 1. 1. 2 One of the following statements should be true for each terminator on the context diagram. The terminator represents a low-level physical interface (i. e. , an input/output device with little or no potential for concurrent access), or The terminator represents a software system satisfying requirements not allocated to your software configuration item (SCI), but with which your SCI is required to interact. No hardware interface is needed to connect your SCI with the other software system. Planning for and Managing VV QA POC J. Lewis 17

Detailed Design Review Checklist (Example) • Has the development team demonstrated that every requirement is traceable to the detailed design? • Is the detailed design valid (complete, consistent, feasible, maintainable, testable? • Are the detailed design and critical parameter budgets (e. g. , sizing, timing, accuracy) for the program within the limits given in the System Specification, and additionally, within the limitations of the physical and functional environments? • Are current, detailed implementation plans and test plans adequate and complete? • Are there any critical issues (e. g. , any requirements not satisfied), and what is required to resolve these issues? Planning for and Managing VV QA POC J. Lewis 18

Indicators (Example) Planning for and Managing VV QA POC J. Lewis 19

Metrics (Example) Average = 5. 7 weeks Target < 8 weeks Planning for and Managing VV QA POC J. Lewis 20

Project Oversight Planning for and Managing VV QA POC J. Lewis 21

Project Oversight Framework • Project oversight is defined as “an independent review and analysis to determine if the project is on track to be completed within the estimated schedule and cost, and will provide the functionality required by the sponsoring business entity. Project oversight identifies and quantifies any issues and risks affecting these project components. ” • Related Terms – V&V – Graduated Oversight – Review , Assess, Report, Track Planning for and Managing VV QA POC J. Lewis 22

Project Oversight • 3 fundamental types – Technical Project Oversight • V&V (IEEE Std 1012 -1998) • QA of Technical Performance • QA of Technical Product – Management Project Oversight • QA of Project Management Function – Processes & Products of Project Management • Project Oversight Framework – Special Project Reporting • In response to external stakeholder – e. g. Sponsor, Control Agency Planning for and Managing VV QA POC J. Lewis 23

Level of Independence Planning for and Managing VV QA POC J. Lewis 24

Level (Degree) of Independence IEEE Standard 1012 -1998 • No Independence – Part of the development organization • Managerial Independence – Separate from the development and program management organizations – Selects the segments of the project/product to subject to oversight • Technical Independence – Utilizes personnel who are not involved in the development of the software • Financial Independence – Control of the oversight budget is vested outside of the development organization Planning for and Managing VV QA POC J. Lewis 25

Software Integrity Levels Planning for and Managing VV QA POC J. Lewis 26

Software Integrity Levels IEEE Standard 1012 -1998 • Integrity levels are – used as an objective basis for quantifying product risk – mandatory (any scheme may be used) • A good example is shown in Annex B of IEEE Std 1012 -1998 (A Software Integrity Level Scheme) – updated at each life-cycle phase (not static) • as part of criticality analysis task – established at the beginning of the project • documented in the project SVVP – used as the basis of V&V planning • assigned to each function (requirement, component, function, etc. ) – documented in a V&V Task Report and the V&V Final Report • adjusted based on developer methods and risk plans Planning for and Managing VV QA POC J. Lewis 27

Example Software Integrity Level Scheme • Based on intended use and application of the software product • Integrity levels are used to quantify software product criticality Planning for and Managing VV QA POC J. Lewis 28

Failure Consequences CONSEQUENCE CATASTROPHIC DESCRIPTION Loss of human life; complete mission failure; loss of system security; extensive financial or social loss CRITICAL Major or permanent injury; partial loss of mission; major financial or social loss MARGINAL Severe injury or illness; degradation of secondary mission; some financial or social loss NEGLIGIBLE Planning for and Managing VV QA POC J. Lewis Minor injury or illness; minor impact on system performance; operator inconvenience 29

Software Integrity Levels Key Features • Software Integrity Levels… Objective Scalability – – – Failure Consequences Intended Use of Software V&V Rigor to be Applied Minimum V&V Tasks via Integrity Levels Defines exit criteria for each task • Levels of V&V Independence • Optional V&V Tasks Planning for and Managing VV QA POC J. Lewis 30

Example – Integrity Level Assignment Integrity Levels 1 3 3 2 1 1 2 3 2 2 2 2 = = = High Major Moderate Low None Notes: • Assigned to all functions • Entire solution receives integrity level of highest sub-function • NOT a static assignment – Only point-in-time based on present information • Recorded in V&V Task Report 3 Planning for and Managing VV QA POC J. Lewis 4 3 2 1 0 31

Integrity Level Assignment… Through the Project Lifecycle Concept e. g. in Con. Ops or FSR 1 3 3 2 1 1 1 2 3 3 2 2 2 3 2 1 1 2 3 2 1 1 2 2 1 3 2 2 O&M 2 3 3 2 2 e. g. In Ops Doc’s 3 2 1 1 2 3 2 2 3 1 2 3 3 2 2 2 3 2 1 1 2 3 2 2 2 3 Planning for and Managing VV QA POC J. Lewis 32 2

Comparison of V&V QA Oversight Planning for and Managing VV QA POC J. Lewis 33

Related of Terms & Concepts Project Oversight V&V • Graduated Oversight • Integrity Levels – Project Complexity – Product Failure Consequences • Independence – From procuring/managing department Planning for and Managing VV QA POC J. Lewis – From developer 34

IV&V vs QA • IV&V – Occurs in parallel to development activities • Part of design team is optimal interface point Ø Independent systems engineering analysis of methods and products • Systems engineering context – Verified compliance with IEEE Standard 1012 -1998 • Quality Assurance – Product evaluation • Compliance with standards, outlines, documentation guidelines Ø Process compliance audit • Repeatability, measure of compliance – Usually within development organizational structure – Compliant with IEEE Standard 730 (QA Plans) Planning for and Managing VV QA POC J. Lewis 35

IV&V vs Project Oversight • IV&V – Systems engineering focus Ø Objective to build quality into system solution during development – Involvement is based on assigned integrity level of product – Degree of independence • Project Oversight – Focused on risk reporting & special studies Ø Objective is to mitigate risk through reporting to higher level – Lower level of involvement • Monthly, Quarterly – Degree of independence Planning for and Managing VV QA POC J. Lewis 36

IV&V vs Project Management Oversight • IV&V Ø Systems engineering focus – Focus on fitness for use of final product • Project Management Oversight Ø Project management focus • Is the project following defined project management processes? – Focus on cost and schedule compliance Planning for and Managing VV QA POC J. Lewis 37

Scoping & Planning V&V QA Oversight Planning for and Managing VV QA POC J. Lewis 38

How to Scope and Cost Technical Oversight Assign / Adopt the software integrity level scheme and. . . Integrity Assessment Risk Assessment Perform analysis to each function of the solution Document Management Risks etc (SEI Risk Taxonomy) Select additional V&V tasks Assign minimum V&V tasks (more minimum tasks + optional tasks) in response to project risk areas according to software integrity level assignments Estimate V&V tasks according to rigor necessary - apply costs via rate structure and/or contract definition Planning for and Managing VV QA POC J. Lewis 39

Minimum V&V Tasks Assigned by Integrity Level • Level 4 – High – All V&V tasks are applied • Level 3 – Major • Level 2 – Moderate • Level 1 – Low – Minimum tasks focus on classical risk areas Planning for and Managing VV QA POC J. Lewis 40

How to Scope and Cost Project Oversight • Determine Scope of Project Oversight – Oversight tasks are defined commensurate with the complexity of the project • Oversight Framework – QA tasks are defined in the QA Plan, defined in response to the suite of management plans created for the project • Determine Cost of Project Oversight – Apply costs via rate structure or contract definition Planning for and Managing VV QA POC J. Lewis 41

How to Scope and Cost Quality Assurance • Develop Management & Technical Plans (Predecessor) • Define Quality Assurance Methods Commensurate with Management and Technical Plans – Methods • Product Conformance • Process Compliance – Reporting (Including Degree of Independence) • Establish Estimates From Work Breakdown – Number of and Timing for Products – Management and Technical Process Model Planning for and Managing VV QA POC J. Lewis 42

Managing V&V QA Oversight Planning for and Managing VV QA POC J. Lewis 43

Considerations for Managing V&V, QA, and Oversight • The oversight entity (entities) should be placed high enough in the organization to be – Free from undue influence – Reporting to a decision maker • In order to be effective, the oversight entity (entities) must be tasked in a manner timely to the process to be able to influence the quality of the results of the process • The oversight entity (entities) must possess the knowledge, skills, and abilities to perform its tasks effectively and efficiently Planning for and Managing VV QA POC J. Lewis 44

Summary Planning for and Managing VV QA POC J. Lewis 45

Summary • V&V, QA, & Oversight – – Are similar Can overlap Evaluate technical and/or management products/performance Should provide decision makers with the right information at the right time to enable informed decision making Planning for and Managing VV QA POC J. Lewis 46

Questions ? Planning for and Managing VV QA POC J. Lewis 47