Fighting Against Software Defects CHEN Xinyu 2010 02

Fighting Against Software Defects CHEN Xinyu 2010 -02 -01 Dept. of Computer Science & Engineering, The Chinese University of Hong Kong

r r r 2 Outline What software defect are Why software defects occur What can be done n n n r Dept. of Computer Science & Engineering, The Chinese University of Hong Kong Detection Analysis Prevention Conclusion 2 Dept. of Computer Science & Engineering, The Chinese University of Hong Kong

Outline r r r What software defects are Why software defects occur What can be done n n n r Detection Analysis Prevention Conclusion Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 3

1. What Software Defects Are Software Defect A software defect is a deficiency in a software product that causes it to perform unexpectedly. Number of Defects Are we loosing control? Minimizing the number of defects We need more information! Software’s complexity and accelerated development schedules make avoiding defects difficult http: //www. research. ibm. com/softeng/ODCEG. HTM Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 4

2. Why Software Defects Occur r A defect in software results from some type of mistake n n Human error Systemic error in the development process Ú Ú r Guidelines and procedures Domain-specific information Organization information … Not every mistake leads to a defect, but almost all defects can be traced back to some type of mistake Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 5

Dynamics of Software Failure Production Requirements Specifications Functionality Semantics … Programmer Knowledge Attention Strategies … Runtime Failures stem Programming Sy Interfaces Problematic Specifications Cognitive Breakdowns Information Notations … Program Usability Issues A. J. Ko, B. A. Myers A frameworkand methodology for studying the causes of software errors in programming systems Journal of Visual Languages & Computing, 16, pp. 41– 84, 2005. Software Defects Dept. of Computer Science & Engineering, The Chinese University of Hong Kong Algorithms Data structures Variables Classes Interfaces … Runtime Errors 6

3. What Can Be Done r Three Levels in Improving Software Quality Learn from the past Proactively identify and eliminate potential defects Look for trends and insights Test quality into the software Dept. of Computer Science & Engineering, The Chinese University of Hong Kong Prevention Analysis Detection 7

A Bug's Life at Microsoft Run test case Product Code issue Design or code Design issue Enter bug report Specify code fix Close bug Proposed code fix Modify design Design OK? Resolve “Won’t Fix” Proposed design fix Not approved No or refine Investigation approved Close bug Pass Regress bug Sustained Engineering Transition Checklist Triage Fix approved Fail A. Page, K. Johnston, B. Rollison, How we test software at Microsoft press, 2009 Reactive bug Dept. of Computer Science & Engineering, The Chinese University of Hong Kong Build workflow Resolve fixed bug Fix code 8

IBM: Orthogonal Defect Classification (ODC) r r Defects are collected analyzed in-process during an on-going software development Classifying each defect into orthogonal (mutually exclusive) attributes Opener section: These are the attributes one can classify when you find a defect ODC Attributes A. A. Shenvi Defect prevention with orthogonal defect classification ISEC’ 09, pp. 83 -87, Feb. 2009 Closer section: These are the attributes one can classify when you know how the defect was fixed Dept. of Computer Science & Engineering, The Chinese University of Hong Kong • Activity • Trigger • Impact • Target • Type • Qualifier • Source • Age 9

Origin (Where? ) HP: Defect Origins, Types, and Modes Specifications/ Requirements Design Code Requirements or Specifications HW Interface Environmental Support Documentation Other Logic Test SW SW Interface Process (Interprocess) Communication Computation Test HW Functionality User Interface Data Definition Data Handling Other Functional Description Module Design Type (What? ) Other Module Interface / Implementation Development Tools Logic Description Error Checking Standards Integration SW Other Mode (Why? ) Missing Unclear Wrong Dept. of Computer Science & Engineering, The Chinese University of Hong Kong Changed Better Way 10

Defect Analysis Example Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 11

Root Cause Analysis r Identify where in the software development lifecycle a defect was introduced n Identify where attention could be concentrated Domain Knowledge Scattered Requirements Misunderstood Requirements Functional Defects Implicit Requirements Not Addressed Missing/Incorrect Requirements/Design Fishbone or cause-and-effect diagram Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 12

Defect Prevention Software Defect Prevention A strategy applied to the software development life cycle that identifies root causes of defects and prevents them from recurring. r A level 5 Key Process Area (KPA) in the Capability Maturity Model Integration (CMMI) Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 13

Work product Fan-out Phenomenon in Software Development Customer Testing Internal Testing Implementation Design Specification Scenarios Time Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 14

Rate of Discovery Moving Quality Upstream Late defect discovery results in significant correct and rework Defect Prevention Old New Requirements Design & Implementation Test Maintenance Time 100 X Increase in Cost of Removing Defects Software Defect Rate of Discovery vs. Time P. Narayana Software defect orevention - In a nutshell http: //software. isixsigma. com/library/content/c 030611 a. asp Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 15

Failure Modes and Effects Analysis (FMEA) r Identify potential failure modes in a product design n A failure mode is a type of failure that can occur in a product How can the system fail? r What bad things will happen if the failure occurs? Assess the risk of each potential failure How many failures will the system experience? r Implement appropriate actions to eliminate or mitigate those failure modes Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 16

Risk Priority Improvement Owner 2 30 Add to code review list Dan 2 20 Improve the API spec. Error message 2 20 Add disk space check 3 Error message 2 18 Create more detailed API doc Mary 2 Stop (exit) setup 1 10 Improve the API spec. John Likelihood Rank Detectability Rank FMEA Worksheet Process Step, Function or Task Potential Failure Mode Potential Effect(s) of Failure Setup phase Fail to create log file dir Fatal error that stops setup 5 Dev oversight 3 Code review Setup phase Fail to create log file dir Fatal error that stops setup 5 API failure 2 Error message Setup phase Fail to create log file dir Fatal error that stops setup 5 No disk space 2 Setup phase Incorrect determine phase Unpredictable errors are generated 3 Incorrect use of API Setup phase Incorrect determine phase Try to reinitialize drive 5 API failure Potential Causes Current Control Setup logging … Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 17 Sam John Impact Rank Risk Priority Number (RPN) = Impact Severity Occurrence Likelihood Detectability

Fault Tree Analysis (FTA) r r r Starts with a failure Focuses on deducing all the potential causes and their relationships using Boolean logic FMEA and FTA are complementary techniques n n FMEA is used to identify potential failures FTA is then used to discover the causes of those failures Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 18

FTA Example Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 19

4. Conclusion r The causes of software defects n n r Human errors Systemic errors Three levels to improve software quality n Defect detection Ú n Defect analysis Ú Ú Ú n Testing ODC Defects origins, types, and modes Root cause analysis Defect prevention Ú Ú FMEA FTA Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 20

Reference r M. Mc. Donald, R. Musson, R. Smith, The practical guide to defect prevention, Microsoft press, 2008 Thank You! Dept. of Computer Science & Engineering, The Chinese University of Hong Kong 21