Software Unit Test Coverage And Test Adequacy 1

Software Unit Test Coverage And Test Adequacy 1

Goal � � � Survey recent research on software test criteria in the last two decades Put it into a uniform framework Address issues like � Defining a test criterion � Understanding different test adequacy criteria � Whether they are worth the cost � Future directions for this subject 2

IMPORTANCE TODAY � Critical feature in the software development cycle � Identify correctness, completeness Quality assurance Verification and validation Reliability estimation � � � http: //www. aptest. com/resources. html 3

PREVIOUS WORK-TEST ADEQUACY A software test adequacy criterion is a predicate that defines what properties of a program must be exercised to constitute a thorough test. Proposed test criteria : Reliability- producing consistent results Validity – producing meaningful results Not practicably applicable Reliability and validity closely related and not independent. 4

Current Work-Test Adequacy Notions associated with a test adequacy criterion � It is a stopping rule: � Determines if sufficient testing is done that it can stop. � E. g. execution of all statements for statement coverage � It provides measurements of test quality: � Degree of adequacy � E. g. Percent of code coverage 5

Uses of Test Adequacy Criteria � Specify a software testing requirement � Determine test cases to satisfy requirement � Determine observations that should be made during testing � Control the cost of testing � Avoid redundant and unnecessary tests � Help assess software dependability � Build confidence in the integrity estimate 6

Categories of Adequacy Criteria Based on: Source of information to specify testing requirements � Specification based � Program based � Combined � Interface based testing approach � Structural � Fault based � Error based 7

STRUCTURAL TESTING CRITERIA Structural Testing Program-Based Control Flow Based Statement Branch Specification-Based Data Flow Based Path Dependence coverage Mutation 8

CONTROL FLOW – BASED TESTING Statement Coverage * Execute every statement at least once * The simplest, weakest type of testing int * p = NULL; if (condition) { p = &variable; *p = 1; } *p = 0; void function(const char* string 1, const char* string 2) { if (condition || strcmp(string 1, string 2) == 0). . . } 9

CONTROL FLOW-BASED TESTING Branch Coverage * Exercise all control statements in the program * Sub statement coverage * All combinations of control transfers may not be checked if (condition 1 && (condition 2 || function 1( ))) statement 1; else statement 2; 10

CONTROL FLOW-BASED TESTING Path Coverage * Execute all paths from program’s entry to exit * Sub branch coverage Number of paths is exponential to the number of branches Many paths are impossible to exercise if (success) statement 1; statement 2; if (success) statement 3; 11

CONTROL FLOW-BASED TESTING Criteria Inclusion Hierarchy Path Coverage Branch Coverage Statement Coverage 12

DATA FLOW-BASED TESTING Structural Testing Program-Based Control Flow Based Specification-Based Data Flow Based Simple Definition-Use Coverage Dependence coverage Testing for structured and dynamic data Inter-procedural testing 13

DATA FLOW-BASED TESTING � Definition Occurrence – Where a value is bound to the variable. (x=1) � Use Occurrence. Where the value is referred (a=x) * C-use : if used to compute a value or as an output (n=x+1) Global c-use: Value is assigned in a block other than the one in which it is being used. Local c-use * P-use: if used to decide whether a predicate is true (x>0) � Definition. Clear path w. r. t x: for all nodes in the path, there is no definition occurance of x 14

Data flow-based testing SIMPLE DEFINITION USE COVERAGE If variable x has a global definition in node i, � All definitions Criterion : test data covers some path which goes from i to a node or edge at which the value assigned to x in node i is used � All Uses Criterion : test data covers at least one path which goes from i to each node and edge at which the value assigned to x in node i is used � All-du-paths: test data covers all paths which go from i to each node and edge at which the value assigned to x in node i is used as exercised There may be infinitely many such paths 15

DATA FLOW-BASED TESTING 1 2 3 4 5 7 6 8 9 16

DATA FLOW-BASED TESTING Criteria Inclusion Hierarchy All-paths All-du-paths All-uses All-c/some-p-uses All-p/some-c-uses All-p-uses All-definitions All-c-uses 17

Data flow-based testing FOR STRUCTURED AND DYNAMIC DATA � Limitations of testing criteria seen so far: No distinction between atomic (integers) and structured data arrays and records) A[0] = A[i]; A[i] = A[j]; A[j]= A[0]; There is a path from each statement to the next. The actual path should involve A [0] in the first and last statement. � Proposed solution: Treat structured data elements as independent data May lead to infinite paths to be tested. 18

Data flow-based testing INTER-PROCEDURAL DATA-FLOW TESTING Test data-dependence among procedure interfaces Types of data dependence � Direct Definition occurs in procedure P, use in directly called procedure Q � Indirect Considers multiple levels of calls and returns Definition occurs in P , use in G. Example: Class Q extends P, R extends Q f(P) f(Q) f(R) 19

FAULT-BASED TESTING CRITERIA Error Seeding � Estimate the number of faults that remain � Measure quality of software testing r= # artificial faults detected # of artificial faults f = # of not seeded errors detected Estimated no. of inherent faults = (1/r)*f � Applicable to any testing method � Dependent on how faults are introduced 20

FAULT-BASED TESTING CRITERIA Program Mutation Testing Mutant: A program with a planted fault � Execute mutants on each member of test set � Compare results � Mutation Adequacy Score = D/N D=No. of dead mutants N = No. of non equivalent mutants c = a + b; R 1 c = a – b; R 2 If (R 1 = R 2): mutant is alive otherwise it is killed. 21

FAULT-BASED TESTING CRITERIA Variants of Program Mutation Testing � Weak Mutation Testing � proposed to improve efficiency � mutate and test components � Firm Mutation Testing � Select portion of program , subset of parameters and mutate them. � Compare original and changed versions � Less expensive than strong mutation testing, more efficient than weak mutation testing � No basis to select area of program code, parameters 22

FAULT-BASED TESTING CRITERIA Criteria Inclusion Hierarchy Strong Mutation Testing Firm Mutation Testing Weak Mutation Testing 23

PERTURBATION TESTING Fault-based testing criteria Tests the robustness of a program � Predicted fault tolerance = # of faults detected total # of executions � A perturbation function is applied to change the data state Example: int perturbation (int x) { int changed. X; changed. X = x + 50; return changed. X; } � 24

FAULT-BASED TESTING CRITERIA Perturbation Testing Original program main() {int x; x = get. Val(); if (x > 0) printf(“X is positive”); else printf(“X is negative”); } Fault injected program main() {int x; x = get. Val(); x = perturbation(x); if (x > 0) printf(“X is positive”); else printf(“X is negative”); } 25

Summary All-paths All-du-paths Strong Mutation Testing All-uses Firm Mutation Testing All-c/some-p-uses All-p/some-c-uses Weak Mutation Testing All-p-uses All-definitions All-c-uses Branch Coverage Statement Coverage 26