An Internally Replicated Quasi Experimental Comparison of Checklist

An Internally Replicated Quasi. Experimental Comparison of Checklist and Perspective-Based Reading of Code Documents Laitenberger, etal IEEE TOSE May 2001 f 02 laitenberger 6 1

Overview - What did you learn? u Inspections u Experimentation f 02 laitenberger 6 2

Inspections SINCE Fagan's initial work presented in 1976 [24], software inspection has emerged in software engineering as one of the most effective and efficient methods for software quality improvement. A software inspection usually consists of several activities including planning, defect detection, defect collection, and defect correction [53]. 3 Inspection planning is performed by an organizer who schedules all subsequent inspection activities. The defect detection and defect collection activities can be f 02 laitenberger 6 performed either by inspectors individually or in a group 3 meeting.

What is PBR Reading? f 02 laitenberger 6 4

Exp Context Guidelines – C 2: If a specific hypothesis is being tested, state it clearly prior to performing the study and discuss theory from which it is derived, so that its implications are apparent. Recent empirical findings reveal that the synergy effect of inspection meetings is rather low in terms of impact on defects detected [55], [85], [44]. Therefore, defect detection can be considered as an individual rather than a group activity. the effectiveness 5 of PBR and its cost per defect ratio are compared with those of CBR. f 02 laitenberger 6 5

Conduct and collect – DC 1: Define all software measures fully, including the entity, attribute, unit and counting rules. 5. In this study, effectiveness is defined as the proportion of all defects in the code that were found by applying one reading technique. f 02 laitenberger 6 6

Exp Context C 1: Be sure to specify as much of the industrial context as possible. In particular, clearly define the entities, attributes, and measures that are capturing the contextual information. f 02 laitenberger 6 7

Exp Design Guidelines - TTYP D 1: Identify the population from which the subjects and objects are drawn. D 2: Define the process by which the subjects and objects were selected. D 3: Define the process by which subjects and objects are assig to treatments. f 02 laitenberger 6 8

Evaluation of Experiment f 02 laitenberger 6 9

Exp Design Guidelines – class D 4: Restrict yourself to simple study designs or, at least, to designs that are fully analyzed in the statistical literature. If you are not using a well-documented design and analysis method, you should consult a statistician to see whether yours is the most effective design for what you want to accomplish. f 02 laitenberger 6 10

Conduct and collect – DC 3? DC 3: Describe any quality control method used to ensure completeness and accuracy of data collection. f 02 laitenberger 6 11

Conduct and collect – DC 5 and 6? DC 5: For observational studies and experiments, record data about subjects who drop out from the studies. DC 6: For observational studies and experiments, record data about other performance measures that may be affected by the treatment, even if they are not the main focus of the study. f 02 laitenberger 6 12

Analysis A 1: Specify any procedures used to control for multiple testing A 2: Consider using blind analysis. A 3: Perform sensitivity analyses. f 02 laitenberger 6 13

Analysis A 4: Ensure that the data do not violate the assumptions of the tests used on them. A 5: Apply appropriate quality control procedures to verify your results. A 3: Perform sensitivity analyses. f 02 laitenberger 6 14

Presentation P 1: Describe or cite a reference for all statistical procedures use P 2: Report the statistical package used. P 3: Present quantitative results as well as significance levels. Quantitative results should show the magnitude of effects and the confidence limits. f 02 laitenberger 6 15

Presentation P 4: Present the raw data whenever possible. Otherwise, confirm that they are available for confidential review by the reviewers and independent auditors. P 5: Provide appropriate descriptive statistics. P 6: Make appropriate use of graphics. f 02 laitenberger 6 16

Interpretation I 1: Define the population to which inferential statistics and predictive models apply. I 2: Differentiate between statistical significance and practical importance. I 3: Define the type of study. I 4: Specify any limitations of the study. f 02 laitenberger 6 17

Results? f 02 laitenberger 6 18

For L 7 (Tues 9/17) u. L Prechelt and B Unger, “An experiment measuring the effects of Personal Software Process (PSP) Training”, IEEE TOSE may 2000 pp 465 -479 f 02 laitenberger 6 19