IVV Facility IVV Transition Concept Review December 12
IV&V Facility IV&V Transition Concept Review December 12, 2003 12/12/2003 IV&V Concept Review 1
Introduction IV&V Facility • Transition Team Members – – – – Steve Raque, IV&V Facility, Lead Neal Barthelme, GSFC Paul Boldon, HQ Code Q Ken Costello, IV&V Facility Rick Grammier, JPL John Kelly, HQ Code AE Significant contributions also provided by Burt Sigal (JPL) and Sue Sekira (GSFC) • Agenda – – 12/12/2003 Concept Review Purpose Background – IV&V Assessment Transition Plan Concept for NASA IV&V Concept Review 2
Concept Review Purpose IV&V Facility • The concept for Agency IV&V is a “system” composed of several “subsystems” (processes) that will be owned by different organizations – An Agency approach to IV&V requires the evaluation and management of software risk across the Enterprises, OSMA, and Engineering • Think of the Concept Review as the “System PDR” of the IV&V Transition • The Concept Review is: – An important step in establishing IV&V as an Agency Program by FY 05 – Enables the establishment of a PCA between GSFC and Code Q – A gate to effecting the transition • Draft Transition Plan and Draft PCA reviewed by GSFC and Code Q • After Concept Review, the Transition Plan and PCA will be formally transmitted to Code Q for approval • The Transition Plan names the AA/OSMA as the overall approving official for the IV&V Concept with the responsibility to call upon other stakeholders and process owners to ensure the concept is accepted Agency-wide. Mr. O’Connor has delegated the review to the GSFC PMC 12/12/2003 IV&V Concept Review 3
Background IV&V Facility • In February 2003, the NASA Associate Deputy Administrator for Technology Programs requested Code Q to lead a team to assess the Agency’s IV&V capability – The assessment team was made up of personnel from HQ, GSFC, JPL, KSC, and DCMA – The assessment was conducted during the months of February and March 2003 – The goals of the assessment were the following: • • • to better understand IV&V's potential to help shape IV&V’s continued role at NASA to determine how best to structure the reporting and support of the IV&V Facility to assure its technical, managerial and financial independence – The team reviewed IV&V’s processes and products. The assessment included interviews with both IV&V customers and with IV&V management. – The final assessment team report was published on April 23, 2003 – The conclusions and recommendations of the assessment team were briefed to the NASA Associate Deputy Administrator for Technology Programs, the NASA Chief Engineer, the Associate Administrator for Safety and Mission Assurance, and the Director of the IV&V Facility on April 4, 2003. At the conclusion of this briefing, the Director of the IV&V Facility was directed to evaluate the assessment team’s conclusions and recommendations and prepare a transition plan for implementation of agreed to changes based on the assessment team’s report. • • On May 13, 2003, the NASA Executive Council was briefed on the results of the assessment and the recommended IV&V management and funding changes During this meeting, and in September of 2003, the Executive Council decided that IV&V would become a Code Q Program delegated to GSFC, an IV&V Board of Directors (IBD) would be formed, IV&V would be funded from Corporate G&A, and measures to improve the effectiveness of IV&V would be put in place 12/12/2003 IV&V Concept Review 4
Transition Plan IV&V Facility • NASA Executive Council decisions are used as the transition goals – – Transition IV&V to a Code Q Program Delegated to GSFC Create an IV&V Board of Directors Transition IV&V Funding to Agency G&A Improve the effectiveness of IV&V processes • The plan objectives are to describe the tasks, responsibilities, participants, schedule, and dependencies that will ensure: – A smooth transition and the goals are met – The recommendations and concerns identified by the Assessment Team are addressed – There is coordination with changes necessary to address IG-03 -011 • AE and Q establish a process to provide IV&V with complete list of NPG 7120. 5 and NTS 8719. 13 projects on a recurring basis • AE verify that programs and projects comply with IV&V policy • The Inspector General (IG) has communicated to the Chief Engineer that IG 03 -011 will be closed. “The IG has determined that the actions taken by NASA (including this transition plan) are sufficient to disposition all of the audit's recommendations. ” (Dec 5, 2003) 12/12/2003 IV&V Concept Review 5
Transition Task/Schedule Overview IV&V Facility 12/12/2003 IV&V Concept Review 6
Key Tasks IV&V Facility 1. Develop Process for Compiling and Validating List of IV&V Candidate Projects (AE owns process) 2. Develop Process for Scoping and Costing IV&V (IV&V owns process) 3. Develop IV&V Implementation Process (IV&V owns process) 4. Define Review, Process Control, and Reporting of IV&V (Code Q and GSFC own processes) 5. Determine Relationships among IV&V, GSFC, HQ, Enterprises, Centers, and Projects The primary goal of the processes developed in tasks 3 and 4 is to improve the effectiveness of IV&V 12/12/2003 IV&V Concept Review 7
IV&V Facility Concept for Agency IV&V 12/12/2003 IV&V Concept Review 8
Overall IV&V Process IV&V Facility 12/12/2003 IV&V Concept Review 9
Ranked List of Projects Based on Criticality of Software IV&V Facility • Process owned by Code AE • Use the knowledge base of Headquarters and Centers to compile a list of Projects with software under development or maintenance governed by NPG 7120. 5 or 8719 – Objective is to cover a five year horizon – Projects are ranked using an Agency-wide criticality criteria • Projects are divided into four major classes – Human Rated (H), Robotic (R), Instruments (I), and Information or Data Analysis systems (D) • Each project is given a score based on the characteristics of the project. Engineering notes can provide mitigations to the score or ranking. 12/12/2003 IV&V Concept Review 10
Annual Process for Compiling and Validating Ranked List of Projects IV&V Facility A. 1 – A. 3 Agency Projects List (B/Q/AE) C. 1 – C. 2 Integrated Ranked list of Candidate Projects (AE) C 3. Agency Review A. 4 AE Requests to Center generate prioritized project lists B. 1 Centers generates prioritized project lists SW Criticality Criteria Worksheet (new) 12/12/2003 SSC JPL KSC La. RC MSFC JSC GSFC GRC DRFC ARC C. 4 Final Integrated Ranked list of Candidate Projects (AE) SW Assurance B. 2 Enterprise & Center Review IV&V Concept Review SW Engineering IV&V Estimation & Budget Approval Process 11
IV&V Facility Ranking Process Frequently Asked Questions Q: Can the Centers submit a project in a different rank order than the position indicated by the worksheet “project criticality score”? A: Yes. The worksheet is an aid to promote consistency, not a substitute for engineering judgment. Q: How much of this process do you expect to see in updated NPDs or NPGs? A: The requirement to develop a ranked list of projects containing significant software and the responsible HQ office should be included in an NPD/NPG. Codes AE and Q will provide an appropriate level of direction on the use of the annual ranked list relevant documentation. 12/12/2003 IV&V Concept Review 12
IV&V Facility Ranking Process Frequently Asked Questions Q: Why is a ranked list needed? A: The ranked list provides a manageable funding cut point which assures Agency resource are being wisely used to mitigate risks in its most critical projects. Q: Do you expect this list to have usefulness beyond the IV&V budget process? A: Yes. I believe that the list will raise the awareness of critical software components and help Software Assurance and Engineering focus their efforts. 12/12/2003 IV&V Concept Review 13
Scoping and Costing IV&V Facility 12/12/2003 IV&V Concept Review 14
Scoping and Costing IV&V Facility • Purpose is to provide the IV&V Board of Directors (IBD) with a life-cycle IV&V cost estimate for each candidate project • The IV&V Facility determines the proper level of IV&V support to be applied to each of the projects and generates a cost estimate for each project on the Agency ranked list • Different cost estimation techniques are used depending upon amount of project detail known (schedule, software size, etc. ) • Data is collected to improve the various cost estimation techniques 12/12/2003 IV&V Concept Review 15
Scoping and Costing IV&V Facility 12/12/2003 IV&V Concept Review 16
IV&V Budget Review Process IV&V Facility 12/12/2003 IV&V Concept Review 17
IV&V Facility IV&V Budget Review and Approval Process • IBD is chaired by AA/OSMA and has the Chief Engineer, the CIO, and the AAs for M, N, R, S, U, and Y as voting members. GSFC Center Director and IV&V Facility Director are ex officio members • The IBD determines the appropriate level of funding for software IV&V and selects the Programs and Projects that require support from the IV&V Facility • It is the intent of the IBD to apply the proper level of IV&V on Programs and Projects that have the most critical software development • Lower priority Projects, which cannot be supported within the IBD established budget, will not receive IV&V from Corporate G&A • IBD annually recommends 5 year IV&V budget to Code Q 12/12/2003 IV&V Concept Review 18
IV&V Facility 12/12/2003 IV&V Services and Integration with the Project Life Cycle IV&V Concept Review 19
IV&V and the Project Life-Cycle IV&V Facility System Requirements Review Initial IVVP Signed Baseline IVVP Signed Concept Phase 1. 3 Preliminary Design Review Critical Design Review S/W FQT System Mission Readiness Test Launch Review - IV&V provides support and reports for Project milestones - Technical Analysis Reports document major phases - IVVP is updated to match changes in Project Requirements Phase 1. 4 Design Phase 1. 5 Implementation Phase 1. 6 IV&V Provides Co. FR Test Phase 1. 7 IV&V Management & Planning and Phase Independent Support 1. 1, 1. 2 System Retirement IV&V Final Report Operations Phase & Maintenance Phase 1. 8, 1. 9 Note: numbers correspond to IV&V WBS • Life-cycle IV&V is designed to mesh with the Project schedule and provide timely inputs to mitigate risk • Dialog between the IV&V Facility and the Project must begin before SRR • For most Projects, IV&V ends (and the Final Report is delivered) on or about MRR. Some Projects have extended S/W development post-launch or major upgrades/maintenance (e. g. Shuttle, MER) 12/12/2003 IV&V Concept Review 20
Management and Support Tasks IV&V Facility Project Software Development Life Cycle IV&V Management and Planning 1. 1 • • • Phase Independent Support 1. 2 These activities help to ensure the proper application of IV&V on projects Executed throughout the Software Development Life Cycle Covers development of IV&V Plan and interfaces with projects – The IV&V Plan (IVVP) documents the IV&V approach for the Project. Some sections of the IVVP require Project concurrence (the sections covering schedule, POCs/Interfaces, reports/deliverables to the Project, and items (e. g. software artifacts) needed by IV&V). – Initial IVVP serves to set overall expectations, POCs, etc. It is not a full-blown plan (not enough software detail is usually known). – Interfaces include project suppliers (such as contractors building software for NASA). Proper language must be included in development contracts to provide IV&V access to software artifacts. • Other cross-cutting tasks such as Issue Tracking and Resolution, management review, project review support, etc. 12/12/2003 IV&V Concept Review 21
Primary Life Cycle Tasks IV&V Facility System Requirements Review Concept Phase 1. 3 • • Preliminary Design Review Critical Design Review Requirements Phase 1. 4 Design Phase 1. 5 S/W FQT Implementation Phase 1. 6 Integration System Mission Readiness & Test Review Test Phase 1. 7 These activities form the core of the effort on any project Includes tasks for all life cycles that examine primarily life cycle products – Includes such things as requirements traceability analysis, design architecture analysis, code analysis, test analysis, etc. • Results are documented thoroughly and presented to the project – Issues and risks are tracked through closure (task under management and support). – Technical reports are provided directly to the projects. – Results/issues are communicated immediately. They are not held until formal technical reports. – Best interface for resolving technical issues is between IV&V analysts and development personnel. 12/12/2003 IV&V Concept Review 22
Close Out and Ongoing Analysis IV&V Facility Launch Begin Operations Operational Maintenance System Retirement Maintenance Phase 1. 9 Operations Phase 1. 8 • IV&V provides a final report documenting its findings and results – Supports mission/launch/flight readiness reviews • Effort not always complete at time of launch – Extended missions require after launch support due to software being developed and uploaded during cruise phase • Some missions have extended life times and perform ongoing software maintenance – IVV support mirrors normal life cycle processes but on a smaller level and in an iterative manner – Each planned change is a mini-IV&V project (Space Shuttle) 12/12/2003 IV&V Concept Review 23
IV&V WBS for NASA Missions IV&V Facility • The purposes of the IV&V Work Breakdown Structure are to: – Provide a consistent approach to IV&V across the Agency (similar missions receive the same amount of risk mitigation) – Provide a consistent and comprehensive basis for collection and reporting of metrics – Help Projects anticipate and understand what IV&V will do for them and what will be needed from them • The IV&V WBS was developed using industry standards and previous IV&V on NASA missions as reference – IEEE Std. 1012 -1998 IEEE Standard for Software Verification and Validation – IEEE/EIA 12207. 0 -1996 Standard for Information Technology-Software life cycle processes – Analysis of all missions supported by the IV&V Facility • Baseline WBS Tasks for NASA Missions – The baseline represents a set of tasks that are nominally performed on a NASA mission of the given type (mission types correspond to those used in ranking criteria) – Used as the task set for pre-startup assessment budget estimates – Tasks are tailored based on project characteristics (module/function level size, complexity, reuse, risk, etc. ) as part of IV&V planning and criticality analysis tasks 12/12/2003 IV&V Concept Review 24
IV&V WBS IV&V Facility • The heading legend is as follows: H – Human Rated, R – Robotics, I – Instrument, D – Science and Data Analysis Items with an X are the baseline tasks for that type of mission Tasks not included in baseline for a particular mission (including 1. 10 Optional Tasks) can be performed: • • – As recommended by IV&V to mitigate particular project risks – When paid for by Projects to provide additional risk mitigation – As funded by the IBD or Code Q 1. 1 IV&V Management and Planning H R I D 1. 2 1. 1. 1 Independent Verification and Validation Planning X X 1. 2. 1 1. 1. 2 Baseline Change Assessment X X 1. 1. 3 Management Review of IV&V X X X 1. 1. 4 Final Report Generation X X X 12/12/2003 H R I D Issue and Risk Tracking X X 1. 2. 2 IV&V Tool Support X X X 1. 2. 3 Management and Technical Review Support X X X 1. 2. 4 Criticality Analysis X X 1. 2. 5 Participate in Inspections & Walkthroughs X X Phase Independent Support IV&V Concept Review 25
IV&V WBS (Continued) IV&V Facility 1. 3 Concept Phase H R I D 1. 4 Requirements Phase H R I D 1. 3. 1 Reusability Assessment X X 1. 4. 1 Traceability Analysis X X 1. 3. 2 Distributed Architecture Assessment X X X 1. 4. 2 Software Requirements Evaluation X X 1. 3. 3 System Requirements Review X X 1. 4. 3 Interface Analysis X X 1. 3. 4 Concept Document Evaluation X 1. 4. 4 System Test Plan Analysis X X 1. 3. 5 Software/User Requirements Allocation Analysis X 1. 4. 5 Acceptance Test Plan Analysis X X 1. 3. 6 Traceability Analysis X 12/12/2003 IV&V Concept Review 26
IV&V WBS (Continued) IV&V Facility 1. 5 Design Phase H R I D 1. 6 Implementation Phase H R I D 1. 5. 1 Traceability Analysis X X 1. 6. 1 Traceability Analysis X X 1. 5. 2 Software Design Evaluation X X 1. 6. 2 Source Code and Documentation Evaluation X X 1. 5. 3 Interface Analysis X X 1. 6. 3 Interface Analysis X X 1. 5. 4 Software FQT Plan Analysis X X 1. 6. 4 System Test Case Analysis X X 1. 5. 5 Software Integration Test Plan Analysis X X 1. 6. 5 Software FQT Case Analysis X X 1. 5. 6 Database Analysis X X 1. 6. 6 Software Integration Test Case Analysis X X 1. 5. 7 Component Test Plan Analysis X X 1. 6. 7 Acceptance Test Case Analysis X X 1. 5. 8 Sizing and Timing Analysis X 1. 6. 8 Software Integration Test Procedure Analysis X X 1. 5. 9 Data Flow Analysis X 1. 6. 9 Software Integration Test Results Analysis X X 1. 6. 10 Component Test Case Analysis X X 1. 6. 11 Sizing and Timing Analysis X 1. 6. 12 System Test Procedure Analysis X* 1. 6. 13 Software FQT Procedure Analysis X* *Performed for safety critical functions only 12/12/2003 IV&V Concept Review 27
IV&V WBS (Continued) IV&V Facility 1. 7 Test Phase H R I D 1. 9 Maintenance Phase H 1. 7. 1 Traceability Analysis X X X 1. 9. 1 IVVP Revision Analysis X 1. 7. 2 Software FQT Results Analysis X X 1. 9. 2 Proposed Change Assessment X 1. 7. 3 System Test Results Analysis X X 1. 9. 3 Task Iteration X 1. 7. 4 Simulation Analysis X 1. 9. 4 Anomaly Evaluation X 1. 8 Operations Phase 1. 9. 5 Migration Assessment X 1. 8. 1 Evaluation of New Constraints X 1. 9. 6 Retirement Assessment X 1. 8. 2 Proposed Change Assessment X 1. 8. 3 Operating Procedure Evaluation X 12/12/2003 X IV&V Concept Review R I D 28
IV&V WBS (Continued) IV&V Facility 1. 10 Optional Tasks 1. 10. 1 Acceptance Test Design Analysis 1. 10. 13 Hazard Analysis 1. 10. 2 Acceptance Test Procedure Analysis 1. 10. 14 Independent Risk Assessment 1. 10. 3 Acceptance Test Results Analysis 1. 10. 15 Independent Testing 1. 10. 4 Algorithm Analysis 1. 10. 16 Operational Evaluation 1. 10. 5 Audit Support 1. 10. 17 Performance Monitoring 1. 10. 6 Component Test Design Analysis 1. 10. 18 Project Management Oversight Support 1. 10. 7 Component Test Procedure Analysis 1. 10. 19 Security Assessment 1. 10. 8 Component Test Results Analysis 1. 10. 20 Software FQT Design Analysis 1. 10. 9 Configuration Management Assessment 1. 10. 21 Software Integration Test Design Analysis 1. 10 Cost Analysis 1. 10. 22 System Test Design Analysis 1. 10. 11 Disaster Recovery Plan Assessment 1. 10. 23 Training Documentation Evaluation 1. 10. 12 Feasibility Study Evaluation 1. 10. 24 User Documentation Evaluation 12/12/2003 IV&V Concept Review 29
IV&V Facility Relationship to HQ IV&V Facility • IV&V reports annual performance and receives approved budget from IBD (Chaired by Code Q) • PCA describes Program Authority – AA/OSMA delegates Program to GSFC Center Director – IV&V Facility Director is Program Manager • Facility works with Code Q IV&V Liaison to coordinate IBD budget inputs and performance reporting 12/12/2003 IV&V Concept Review 30
IV&V/Center/Project Relationships IV&V Facility • IV&V-Project Relationship: – IV&V still reports issues to Project first and treats Project as primary “customer” for technical findings and risks – As a Code Q Program, IV&V will keep Center S&MA personnel informed of IV&V technical issues so that S&MA has a complete mission assurance picture • IV&V-Center Relationship: – Center Liaison facilitates the startup of IV&V on new projects – Center Liaison and IV&V Facility Leads facilitate technical issue resolution – Center Liaison promotes consistent approaches to IV&V on Projects, and promotes awareness of IV&V Center-wide – S&MA, Projects, and IV&V provide technical status and issues to the GPMC • IV&V reports to GSFC PMC as a Program Office 12/12/2003 IV&V Concept Review 31
IV&V Reporting Approach IV&V Facility Note – Other channels of communication can be instituted on a Center-by Center basis as needed or required 12/12/2003 IV&V Concept Review 32
Center-level IV&V Issue Escalation Process IV&V Facility Notes: 12/12/2003 1. Center S&MA organizations provide alternate/additional path for issue escalation 2. On a Center-by-Center basis, other individuals and organizations may be brought into this process IV&V Concept Review 33
Summary & Next Steps IV&V Facility • Agency Concept for IV&V fulfills all requirements from IV&V Assessment • Concept approval clears the way for full implementation • Next steps are for each entity that owns the various processes to document those processes to the level of detail necessary for implementation • Submit final Transition Plan and PCA for Approval 12/12/2003 IV&V Concept Review 34
IV&V Facility Backup Material 12/12/2003 IV&V Concept Review 35
Ranking Process Worksheet with examples IV&V Facility Engineering Note: This is a complex and new approach to launching an existing spacecraft with crew which presents above nominal risk. However, these changes do not effect the pre-launch processing (e. g. , component score of 0). 12/12/2003 IV&V Concept Review 36
IV&V Facility Ranking Process Worksheet with examples (continued) Engineering Note: Redundancy management and crew interface are assessed to be at a nominal level of risk and not a significant challenge to the project (0 score for component). Engineering Note: The is a modification to a fighter airplane. New software is going to be processing aero-surface controls in order to achieve better performance. OSMA has assessed the aero-surface control, critical systems control and critical systems monitoring subsystems as being safety critical (Scores shown in Bold Italic). 12/12/2003 IV&V Concept Review 37
IV&V Facility 12/12/2003 Ranking Process Worksheet with examples (continued) IV&V Concept Review 38
IV&V Facility 12/12/2003 Ranking Process Worksheet with examples (continued) IV&V Concept Review 39
- Slides: 39