Software Unit Test Coverage And Test Adequacy Hong

Software Unit Test Coverage And Test Adequacy Hong Zhu, Patrick A. V. Hall, John H. R. May Presented By: Arpita Gandhi 1

Goal l Survey recent research on software test criteria in the last two decades Put it into a uniform framework Address issues like l Defining a test criterion l Understanding different test adequacy criteria l Whether they are worth the cost l Future directions for this subject 2

Importance today l Critical feature in the software development cycle l Identify correctness, completeness Quality assurance Verification and validation Reliability estimation l l l http: //www. aptest. com/resources. html 3

Previous Work-Test Adequacy A software test adequacy criterion is a predicate that defines what properties of a program must be exercised to constitute a thorough test. Proposed test criteria : Reliability- producing consistent results Validity – producing meaningful results Not practicably applicable Reliability and validity closely related and not independent. 4

Current Work-Test Adequacy Notions associated with a test adequacy criterion l It is a stopping rule: l Determines if sufficient testing is done that it can stop. l E. g. execution of all statements for statement coverage l It provides measurements of test quality: l Degree of adequacy l E. g. Percent of code coverage 5

Uses of Test Adequacy Criteria l Specify a software testing requirement l Determine test cases to satisfy requirement l Determine observations that should be made during testing l Control the cost of testing l Avoid redundant and unnecessary tests l Help assess software dependability l Build confidence in the integrity estimate 6

Categories of Adequacy Criteria Based on: l Source of information to specify testing requirements l Specification based l Program based l Combined l Interface based l Underlying testing approach l Structural l Fault based l Error based 7

Structural Testing Criteria Structural Testing Program-Based Control Flow Based Statement Branch Specification-Based Data Flow Based Path Dependence coverage Mutation 8

Control flow – based testing Statement Coverage l Execute every statement at least once l The simplest , weakest type of testing int* p = NULL; if (condition) { p = &variable; *p = 1; } *p = 0; void function(const char* string 1, const char* string 2) { if (condition || strcmp(string 1, string 2) == 0). . . } Useful link: http: //www. bullseye. com/line. Coverage. html 9

Control flow-based testing Branch Coverage l l l Exercise all control statements in the program Subsumes statement coverage All combinations of control transfers may not be checked if (condition 1 && (condition 2 || function 1( ))) statement 1; else statement 2; 10

Control flow-based testing Path Coverage l Execute all paths from program’s entry to exit l Subsumes branch coverage Number of paths is exponential to the number of branches Many paths are impossible to exercise if (success) statement 1; statement 2; if (success) statement 3; 11

Control flow-based testing Criteria Inclusion Hierarchy Path Coverage Branch Coverage Statement Coverage 12

Data flow-based testing Structural Testing Program-Based Control Flow Based Specification-Based Data Flow Based Simple Definition-Use Coverage Dependence coverage Testing for structured and dynamic data Inter-procedural testing 13

Data flow-based testing l Definition Occurance – Where a value is bound to the variable. (x=1) l Use Occurance- Where the value is referred (a=x) l C-use : if used compute a value or as an output (n=x+1) l Global c-use: Value is assigned in a block other than the one in which it is being used. l Local c-use l P-use: if used to decide whether a predicate is true (x>0) l Definition-Clear path w. r. t x: for all nodes in the path, there is no definition occurance of x 14

Data flow-based testing Simple definition use coverage l l l If variable x has a global definition in node i, All definitions Criterion : test data covers some path which goes from i to a node or edge at which the value assigned to x in node i is used All Uses Criterion : test data covers at least one path which goes from i to each node and edge at which the value assigned to x in node i is used All-du-paths: test data covers all paths which go from i to each node and edge at which the value assigned to x in node i is used are exercised There may be infinitely many such paths 15

Data flow-based testing 1 2 3 4 5 7 6 8 9 16

Data flow-based testing Criteria Inclusion Hierarchy All-paths All-du-paths All-uses All-c/some-p-uses All-p/some-c-uses All-p-uses All-definitions All-c-uses 17

Data flow-based testing For structured and dynamic data l Limitations of testing criteria seen so far: No distinction between atomic (integers) and structured data arrays and records) A[0 l = A[il; A[i] = A[j]; A[jl = A[0 l; There is a DU path from each statement to the next. The actual DU path should involve A [0] in the first and last statement. l Proposed solution: Treat structured data elements as independent data May lead to infinite du paths to be tested. 18

Data flow-based testing Inter-procedural data-flow testing Test data-dependence among procedure interfaces Types of data dependence f(P) l Direct l Definition occurs in procedure P, f(Q) use in directly called procedure Q l Indirect f(R) l Considers multiple levels of calls and returns l Definition occurs in P , use in G. Example: Class Q extends P, R extends Q 19

Fault-based testing criteria Error Seeding l Estimate the number of faults that remain l Measure quality of software testing r= # artificial faults detected # of artificial faults f = # of not seeded errors detected Estimated no. of inherent faults = (1/r)*f l Applicable to any testing method l Dependent on how faults are introduced 20

Fault-based testing criteria Program Mutation Testing Mutant: A program with a planted fault l Execute mutants on each member of test set l Compare results l Mutation Adequacy Score =D/N D=No. of dead mutants N = No. of non equivalent mutants c = a + b; R 1 c = a – b; R 2 If (R 1 = R 2): mutant is alive otherwise it is killed. 21

Fault-based testing criteria Variants of Program Mutation Testing l Weak Mutation Testing l proposed to improve efficiency l mutate and test components l Firm Mutation Testing l Select portion of program , subset of parameters and mutate them. l Compare original and changed versions l Less expensive than strong mutation testing, more efficient than weak mutation testing l No basis to select area of program code, parameters 22

Fault-based testing criteria Criteria Inclusion Hierarchy Strong Mutation Testing Firm Mutation Testing Weak Mutation Testing 23

Fault-based testing criteria Perturbation Testing Tests the robustness of a program l Predicted fault tolerance = # of faults detected total # of executions l A perturbation function is applied to change the data state Example: int perturbation (int x) { int changed. X; changed. X = x + 50; return changed. X; } l 24

Fault-based testing criteria Perturbation Testing Original program main() {int x; x = get. Val(); if (x > 0) printf(“X is positive”); else printf(“X is negative”); } Fault injected program main() {int x; x = get. Val(); x = perturbation(x); if (x > 0) printf(“X is positive”); else printf(“X is negative”); } 25

Summary All-paths All-du-paths Strong Mutation Testing All-uses Firm Mutation Testing All-c/some-p-uses All-p/some-c-uses Weak Mutation Testing All-p-uses All-definitions All-c-uses Branch Coverage Statement Coverage 26