Reading Techniques for Software Inspection CSC 8350 Advanced

Reading Techniques for Software Inspection CSC 8350: Advanced Software Engineering Instructor: Dr. Xiaolin Hu Presented by Rufus Oladele and Ngozi Oleleh On Monday, 4/14/08

OUTLINE o Software inspection n Definition n Benefits n Inspection and Testing -Comparison n Inspection process n Challenge o Reading Techniques n n Classes Taxonomy o Case Study o Conclusion o References

Definition Software Inspection …. . o Is a static analysis technique to verify quality properties of software. o Supports structured quality improvement. o Enables defect detection in early stages of software development. o Does not require executable code (applicable to any life cycle product). Inspection procedure follow three steps: 1. Defect detection (individual activity, with reading technique support) -READING 2. Defect collection (team activity to identify false positives)-MEETING 3. Defect repair-REWORK

Benefits o Early defect detection improves product quality and reduces avoidable rework down to 10 -20%[Capers Jones, 1991], and www. cebase. org o Inspection uncovers defects that may not be discovered by testing [Basili&Selby, 1987] o Inspection process improves learning and communication within the software team (Briand et al, 2000) o Overall, inspections n n n reduce the software development cost, increase the quality of software, and improves the productivity as well as the quality of the decision making process for management ((Graham, 1993)

Inspection and Testing -Comparison o o o o They both have the same goal- to raise the quality of the product One typical strength of inspection is that a similar process can be applied to a wide range of documents Another strength of inspection is detection of faults in early stage of development process One advantage with testing can be that it is closer to the way the end-user will use the system There exist areas where testing is the only option and where you can not find the defects with inspection. Example is tests that test if the system has the right quality attributes with performance and stress testing Inspections focuses on finding faults, whereas testing mainly focuses on finding failures (which are the result of one or many faults) They are more compliments to each other than competing methods because they are used to find different faults and in different areas

Inspection and Testing -Comparison o o o The technique which is applied first always seems more effective The best way, according to Edward F. Weller, are to make inspection first and after that the testing The most popular approach though is the other way around, first testing and then inspection. Inspections are not widely used in software industry About 40% of software companies do it (Ciolkowski et al, 2003) , why? n Inspection is labor intensive, the return from it is not immediate and clear to the management (Shirey, 1992) n Perception that it costs more than it is worth (Russell, 1991)

Fagan Inspection Process

Conventional Meeting-Based Inspection Process

Challenge o o Problem n Inspectors often do not know how to read document Reason n Developers learn how to write requirements, design, code, etc. But, they do not learn how to read them. n Little technical support for the reading process is currently available. Solution n Provide well-defined and tailored reading techniques. n Reading techniques are different techniques to be used during inspection, mainly related to the type of guidance the inspectors receive for accomplishing their task. n Reading Techniques have become the most active research area among software inspection researchers (Sami Kollanus, Jussi Koskinen 2007) Benefits n Increase the cost-effectiveness of inspections n Provide models for writing documents of higher quality n Reduce human influence on inspection results (i. e. , ensure a more engineering approach in inspections)

Reading Techniques – Two Broad Classes o Passive Reading Techniques n n Inspectors follow a sequence of individual steps (e. g. a given checklist) and let the inspector figure out how to proceed best. o Active Reading Techniques n n Provide details on the inspections process (how to perform an inspection). Includes a separation of perception (what to inspect), e. g. focus on different defect severity classes, defect types, etc. Provide guidance through the most important parts of the document. Support inspectors in their defect detection process.

Reading Techniques: Taxonomy o Ad-hoc-based reading n No support is given for reading n Support = guidelines, what to check, how to check n reviewers use only their own knowledge and experience to identify defects. o Checklist-based reading (CBR) n a list of questions is provided specifying what properties of the document must be checked and what specific problems should be searched for. n General purpose approach, independent of the application domain. n Application of checklist questions to requirements documents sequentially. n Strongly dependent on inspector capability and domain knowledge. n Considered to be the standard reading technique in software organizations.

Reading Techniques: Taxonomy o Checklist example (Wiegers, 2003) 1. Organization and Completeness o Are all internal cross-references to other requirements correct? o Are all requirements written at a consistent and appropriate level of detail? o Do the requirements provide an adequate basis for design? o Is the implementation priority of each requirement included? 2. Correctness o Do any requirements conflict with or duplicate other requirements? o Is each requirement written in clear, concise, unambiguous language? o Is each requirement verifiable by testing, demonstration, review, or analysis?

Reading Techniques: Taxonomy o o Defect-based reading (DBR) n Geared towards detecting specific types of faults n Defects are categorized and characterized, n A set of questions (called scenario) developed for each defect class to guide the reader. n Some empirical evidence exists that this may out-perform checklist-based and ad-hoc approaches Perspective-based reading (PBR) n Similar to defect-based reading, but instead of defects readers have different roles (tester, designer and user) to guide them in reading. n The PBR scenario is made up of 3 major sections: introduction, instructions, and questions n The introduction describes the quality requirements that are most relevant to this perspective

Reading Techniques: Taxonomy n The instructions describe what kind of documents to use, how to read and extract information n The questions are the set of questions which inspector has to answer during inspection n the objective of PBR is to gain a better defect detection coverage of a software artifact n Experiments conducted at the University of Maryland suggest that defect-based reading is more effective than ad hoc reading. n PBR has been extensively empirically evaluated for requirements documents. n It has also been adapted and empirically evaluated for design inspections, code inspections and usability inspections.

Reading Techniques: Taxonomy o Traceability-based reading (TBR) n n RT adapted for inspecting OO design specifications Divided into vertical and horizontal reading. vertical reading checks whether the design corresponds to the requirements horizontal reading checks the different design artifacts (e. g. , class diagrams, package diagrams, and state machine diagrams) against each other o Usage-based reading (UBR) n n n Recent approach to OO software inspection useful for documents developed after use case derivation, such as requirements, design, code, and test documents It assumes that the use cases and scenarios have been defined earlier in the development process.

Reading Techniques: Taxonomy o Usage-based reading (contd) n n UBR utilizes the set of use cases to focus the inspection effort, just as test cases focus the test effort. Consists of the following five steps 1. Prioritize the use cases in order of importance from a user perspective. Select the use case with the highest priority. Track the use case’s scenarios through the document under inspection. During tracking, ensure that the document under inspection fulfills the use case’s goal; e. g. , make sure it provides the needed functionality and that the interfaces are correct. Identify and report the issues that tracking reveals. 5. Select the next use case and repeat from step 3 until the time is up or you’ve covered all use cases. Regarded as best-practice inspection (Winkler &Biffl 2006) 2. 3. 4. n

Case Study: Experimental comparison of UBR and CBR [Thelin, Runeson, Wohlin, 2003] o o o Goal Experiment Preparation Experiment Planning Result and Observation Discussion

Goal and Experiment Preparation o Goal: - To compare and evaluate how well UBR performs relative to CBR o Experiment Preparation: - Inspection Material n Experiment is based on 4 documents in structured text o o n n 1 requirements document 1 design document written in specification and design language (SDL) 1 use case document 1 checklist Design document was inspected using requirements document as a reference The design consists of software modules of a taxi management system and descriptions of signals between these modules

Goal and Experiment Preparation o Experiment Preparation: - Inspection Material n The design document contains 38 faults, of which 2 are new faults found during the experiment and 8 are seeded faults injected by the person who developed the system. n The 28 others are faults made during development of the design document and later found in inspection or test. n These faults were reinserted prior to the experiment.

The Taxi Management system

Experiment Preparation o Roles and Activities n n n The development of the documents and the design of the experiments involved 6 people in total Development - 1 person Prioritization of use cases- 3 people Classification of faults – 2 people Design and analysis of experiment – 1 person o Reviewers n n n 23 software engineering masters students at a university in Sweden Many have extensive experience in software development Have industrial experience and are comparable to fresh software engineers in the industry

Experiment Preparation o Classification of faults Class A – functions affected are crucial for users Class B – functions affected are important for users Class C – functions affected unimportant for users Design documents contains o 13 class A faults o 14 class B faults o 11 class C faults n No syntax errors were logged as faults n n

Experimental Planning o o Independent Variables n UBR or CBR Dependent Variables n Preparation time n Inspection time n Clock time when each fault were found n Number of faults found by each reviewer n Number of faults found by each experiment group n Efficiency (fault/hour) n Effectiveness (detection rate) o Controlled variables n Experience of the reviewers measured on an ordinal scale

Experimental Planning o Hypotheses n H 0 Eff —No difference in efficiency between the UBR-reviewers and CBR-reviewers n HA Eff —There is a difference in efficiency between the UBRreviewers and CBR-reviewers n H 0 Rate —No difference in effectiveness between the UBRreviewers and CBR-reviewers n HA Rate —There is a difference in effectiveness between the UBRreviewers and CBR-reviewers n H 0 Fault—The UBR-reviewers do not detect different faults the CBR-reviewers n HA Fault—The UBR-reviewers detect different faults than the CBR -reviewers

Experimental Planning - Design o o Students divided randomly into 2 groups – 11 UBR group, 12 CBR group Questionnaire administered to explore students experiences in programming, inspections, SDL, use case and taxi system o The questionnaire breaks down the student testers into less qualified, medium and highly qualified inspectors groups Each group is assigned an equal number all group of testers Experiment instrumented with n 1 requirements document n 1 design document n 1 use case document n 1 checklist n 1 inspection record o o

Results and Observation Mean and Standard Deviation Values for Preparation and Inspection Time (Minutes)

Comparison of efficiency for an average reviewer

A comparison of the effectiveness for an average Reviewer

A comparison of the number of unique faults found

Efficiency for all fault, class A faults, and class A and B faults

Effectiveness for all fault, class A faults, and class A and B faults

Discussion of Results From the tables and graphs of results presented earlier, the following interpretations are obvious. o Efficiency—Reviewers using UBR are significantly more efficient than reviewers using CBR. This difference is significant for all faults and for critical faults. o Effectiveness—Reviewers using UBR are significantly more effective than reviewers using CBR. This difference is significant for critical faults, but not for all faults. o Faults—Reviewers using UBR find different and more unique faults and, especially, more critical faults than reviewers using CBR

Discussion of Results Team analysis result also reveals that: o UBR is more effective and efficient than CBR. This is true for all team sizes ( As long as each team contains a normal distribution of average, medium and experienced Inspectors). o A reviewer applying UBR starts to find faults earlier than a reviewer using CBR. The differences for all faults are about 20 minutes and this difference is even larger for critical faults.

Conclusion o o o Software reading is the core activity in many software engineering tasks: verification and validation, maintenance, evolution, and reuse The research is inconclusive on the question of what an efficient reading technique is, more work is needed. The question of which reading technique should be used for which type of software artifact is still an open problem. The inability to find a superior way of reading may be due to the different reading techniques being suited for different artifacts or even applications. Object orientation, UML, and Java have been major causes of changes in the last couple of years, but it is not clear how to efficiently inspect these models and notations. The question of what is the best way to support an inspectors in their work is not clear. There is some work done on how to develop checklists in an efficient way and also how to best support different scenarios when using scenario-based technique

References [1] T. Thelin, P. Runeson, and C. Wohlin, An Experimental Comparison of Usage. Based and Checklist-Based Reading. IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 29, NO. 8, AUGUST 2003. [2] D. Winkler and S. Biffl, An Empirical Study on Design Quality Improvement from Best-Practice Inspection and Pair Programming. J. Münch and M. Vierimaa (Eds. ): PROFES 2006, LNCS 4034, pp. 319 – 333, 2006. [3] A. Aurum, H. Petersson and C. Wohlin, “State-of-the-art: Software Inspections after 25 years”, Software Testing, Verification and Reliability, Vol. 12, No. 3, pp. 133 -154, 2002. [4] K. Petersen, How Do Use Cases Make Inspections More Efficient and Effective? -Further Experimentation with Usage-Based Software Inspections, M. Sc. Thesis, Blekinge Institute of Technology, Sweden, MSE-2006: 04 , 2006 [5] M. Fagan, Design and Code Inspection To Reduce Errors in Program Development. IBM Systems Journal, 15 (3), 182 -211, 1976.