Preventing and Eliminating Software Faults through the Life

Preventing and Eliminating Software Faults through the Life Cycle PI: Katerina Goseva-Popstojanova Student: Margaret Hamill Lane Dept. Computer Science and Electrical Engineering West Virginia University, Morgantown, WV E-mail: Katerina. Goseva@mail. wvu. edu SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova

Problem • NASA spends time and effort to track problem reports/change data for every project. These repositories are rich, underused sources about the way software systems fail and the software faults that cause these failures. • Our goal: Based on systematic and thorough analysis of the available empirical data, build quantitative and qualitative knowledge that contributes towards improvement of software quality by – preventing introduction of faults into the system – more efficiently eliminating them through the life cycle – compiling lessons learned & recommendations to be used by the pilot project and throughout the Agency SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova 2

Approach Explore change tracking systems & automate data extraction Quantify multiple dimensions for each fault type Identify common patterns and unusual dependencies Propose appropriate classification scheme Refine Classify fault/failure data from the pilot project Compile a check list to support avoidance & elimination of different faults types Identify the most frequent fault Compile Lessons Learned & types & most frequent failure Recommendations document types 3 SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova 3

Pilot study: Basic facts • A large NASA mission consisting of millions of lines of code – 21 CSCIs, over 8, 000 files • Developed iteratively at two different locations – Each CSCI has multiple releases with added functionality • We analyzed over 2, 800 Software Change Requests (SCRs) collected in a period of almost 10 years SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova CSCI id 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 Total # of releases # of files 1 2 2 2 2 3 3 3 3 4 4 5 7 7 -- 207 200 287 228 269 321 289 270 125 356 444 277 599 280 81 169 747 587 552 1368 415 8071 #non conf SCRs 392 24 9 16 13 22 77 107 21 27 43 12 121 83 108 135 201 211 263 861 112 2858 non conf SCRs per file 1. 89 0. 12 0. 03 0. 07 0. 05 0. 07 0. 27 0. 40 0. 17 0. 08 0. 10 0. 04 0. 20 0. 30 1. 33 0. 80 0. 27 0. 36 0. 48 0. 63 0. 27 4 0. 35

Technical challenges • Assuring data quality is an important step of any empirical research effort – Inaccurate data may lead to misleading observations – Both the IV&V team and the project team have been extremely valuable in helping us to understand the change tracking system, determine the meaning of different attributes, and verify the quality of the data • The research approach and analysis techniques can be used by any project that tracks problem reports/change data – However, due to the lack of a unified change tracking system, some amount of unique work on exploration of the data format and automation of data extraction may be needed SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova 5

Current capability • Fault localization [SAS 07] – 58 -62% of failures map to faults in more than one file • Sources of failures (i. e. , type of faults) – Dominating fault types: requirements faults (33%), coding faults (33%), data problems (14%) • Activities when the problem was discovered (e. g. , inspection, testing, analysis, on-orbit) – Almost 50% discovered by analysis, only 3% on-orbit • Severity – Around 8% of SCRs are safety critical, less than 1% on-orbit • Lessons learned & recommendations for product and process improvements – Compiled a document for internal use based on our results and the feedback from the IV&V team and project team SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova 6

Current capability: Identifying the most common fault types Most common fault types – Requirements faults: 33% – Coding faults: 33% – Data problems: 14% Early life cycle activities (requirements & design): 38. 25% Late life cycle activities (coding, interface & integration): 48. 57% This finding contradicts the common belief that majority of faults are introduced 7 during early life cycle activities [Boehm et al. 1975, Endres 1975, Basili et al. 1984] SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova

Current capability: Distribution of fault types Development & testing vs. On-orbit Only 3% of SCRs are on-orbit. Note the logarithmic scale of Y axes. SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova 8

Current capability: Safety critical SCRs across fault types Only around 8% of all SCRs are safety critical. The highest percentage of safety critical SCRs comes from coding faults (3. 60%), followed by requirement faults, design faults & data problems 9 (3. 35% total). SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova

Current capability: Lessons Learned / Recommendations • Based on our results and the feedback from the IV&V team and project team we compiled a document for internal use which summarizes the Lessons learned & Recommendations for product and process improvement – Prevent the introduction of faults and improve the effectiveness of detection • Example: Increase effort spent on design and implementation of data repository used to share data between CSCIs – Improve the quality of the data & change tracking process • Example: Ensure the changes to the software artifacts (e. g. , requirements, code, etc) made to fix the problem are recorded and can be easily associated with a specific SCR SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova 10

Planned capability [FY 08 -10] • Classify faults and failures using several additional attributes • Conduct more complex, multivariable analysis • Continually update Lessons Learned & Recommendations for Improvement document • Explore the best ways to prevent and eliminate most common faults and failures throughout the lifecycle; compile the results in a check list • Increase awareness of our work so other projects within NASA can benefit from it SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova 11

Relevance to NASA • Our research is based on a large NASA mission as a pilot study – The research work is done in active collaboration with the IV&V team and the project team – To the best of our knowledge, this is the largest dataset considered so far in the published literature – Quantitative and qualitative results are regularly delivered and discussed with the project • Based on the internal & external validity of our results, we believe that observed trends are intrinsic characteristics of large, complex software systems – Other similar projects across the Agency may benefit – Lessons Learned & Recommendations can be used proactively by newer projects SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova 12

Acknowledgements We thank the following NASA civil servants and contractors for their valuable support! • • Jill Broadwater Pete Cerna Susan Creasy Randolph Copeland James Dalton Bryan Fritch Nick Guerra John Hinkle SAS_08_Preventing_Eliminating_SWfaults_Goseva-Popstojanova • • Lynda Kelsoe Debbie Miele Lisa Montgomery Don Ohi Chad Pokryzwa David Pruett Timothy Plew Scott Radabaugh 13