Independent Verification and Validation Current Status Challenges and

Independent Verification and Validation Current Status, Challenges, and Research Opportunities Dan Mc. Caugherty IV&V Program Manager Titan Systems Corporation 5 September, 2001 1

Agenda • Where we are today • Developer challenges observed by IV&V • IV&V challenges • Opportunities for R&D 2

Where we are today 3

IV&V Gaining Acceptance • IV&V is gaining acceptance as a mechanism for mitigating software risk • IV&V is naturally evolving to also include programmatic assessments such as cost/schedule performance analysis • Recent NASA IV&V policy presents an opportunity for wider application and understanding of IV&V 4

IT Project Success Factors • Gartner Institute study identified the following essential areas as critical to IT project success 1 – – Risk Management Communications Resource Management Lessons Learned • Additional areas based on professional experience – Project planning – Requirements!!! • Projected that implementation of sound project management approach emphasizing these key areas can reduce number of failed projects by 30%1 1 Gartner. Group 5

IT Project Success Rates • Odds for Success 1&2 – > 80% of IT projects are late, over budget, lack essential functionality, or are never delivered – 28% of IT projects just flat fail representing more than $81 Billion in losses – Of 15, 000 data migration projects started in 1999, as many as 88 percent either overran or failed • Leading causes for project failure 1 – poor project planning – failure to manage scope and change – failure to manage risk 1 Gartner. Group 2 The Standish Group 6

Developer Challenges Observed by IV&V 7

Incremental Delivery Definition • “Build a Little, Test a Little” approach can avoid major development failures, but creates other challenges, such as: – Maintaining end state specifications concurrent with increment specifications – Implementing the right requirements in the right builds to meet testable increment specific capabilities – Understanding the cost/schedule effects of pushing requirements into later builds 8

Early Definition of Interfaces and Key Architectural Concepts • Late interface or architecture changes or problems create significant re-work or are unmanageable • Distributed computing stresses system engineering efforts for early definition of – – API’s, remote procedures, and reused objects Redundancy Management, Fault Detection, Isolation, and Recovery Exception handling 9

Use of New Technologies • A great way to attract and keep highly skilled professionals • Majority of focus on what the technology can do, not enough focus on what it can not do or was never designed to do • Limited skill base for new technologies • If COTS, may not have future support 10

Measuring Progress or Quality • Measurable definitions for “Done” are needed. • Metrics efforts and collection mechanisms need to be part of the process from day one • Optimism (a good thing) must be tempered with facts 11

IV&V Challenges 12

IV&V Technical Challenges • Adapting IV&V methods to developer environments • Understanding system dynamic behavior • Convincing a project that the process is too risky • Validating Non-Development-Items • Reverse engineering incompletely specified systems 13

IV&V Managerial and Business Challenges • Demonstrating that selected IV&V tasks and scope are appropriate for the Project’s acceptable levels of risk • Appropriate IV&V integration into project • Demonstrating IV&V ROI • Accounting for IV&V data needs within developer contracts. 14

Opportunities for R&D 15

Opportunities for R&D • Research IV&V ROI – Past case study 3 shows 1. 25 < IV&V ROI < 1. 82 – Recommend a study of many projects with varying amounts and types of IV&V – Correlate overall project success to IV&V effort types • Continue development of code reverse engineering tools and interface V&V tools – Software and Interface Analysis Tool (SIAT) – Interface Management System (IMS) • Develop Reconfigurable engineering simulation for Space Craft – Improve early “validation” analyses 3 Rogers 16

Opportunities for R&D • Develop a guidebook identifying sets of IV&V tasks for certain types of systems and acceptable levels of mission risk. – Research history of IV&V method application and results • Develop a guidebook for developer managers on how to realize benefits of IV&V. – Contracts and agreements – Integration of IV&V into different types of projects – Reporting and issue resolution 17

Summary 18

Summary • There are many software challenges/risks out there – Many can be managed through use of accepted engineering practices • IV&V can help mitigate risk if used/applied correctly • We need to better quantify the effects of IV&V and the ROI • Recommended areas of R&D – Research IV&V ROI, effects of IV&V use – Develop tools that are needed today – Develop guidance for future IV&V'ers and IV&V users. 19