Software Testing Syntaxbased Testing Syntax Coverage Four Structures

Software Testing Syntax-based Testing

Syntax Coverage Four Structures for Modeling Software Graphs Logic Input Space Syntax Applied to Source FSMs Specs Source Specs Design Use cases Applied to DNF Source Models Integ Input

Applying Syntax-based Testing to Programs • Syntax-based criteria originated with programs and have been used most with programs • BNF criteria are most commonly used to test compilers • Mutation testing criteria are most commonly used for unit testing and integration testing of classes

Instantiating Grammar-Based Testing Program-based 5. 2 Grammar String mutation Integration String mutation • Program mutation • Valid strings • Mutants are not tests • Must kill mutants • Compiler testing • Valid and invalid strings Model-Based Input-Based String mutation • FSMs • Model checking • Valid strings • Traces are tests • Test how classes interact • Valid strings • Mutants are not tests • Must kill mutants • Includes OO Grammar String mutation • Input validation testing • XML and others • Invalid strings • No ground strings • Mutants are tests • Input validation testing • XML and others • Valid strings 4

Program-based Grammars • The original and most widely known application of syntax-based testing is to modify programs • Operators modify a ground string (program under test) to create mutant programs • Mutant programs must compile correctly (valid strings) 5

Program-based Grammars • Mutants are not tests, but used to find tests • Once mutants are defined, tests must be found to cause mutants to fail when executed • This is called “killing mutants” 6

Mutation Testing • Ground string: A string in the grammar –The term “ground” is used as a reference to algebraic ground terms • Mutation Operator : A rule that specifies syntactic variations of strings generated from a grammar • Mutant : The result of one application of a mutation operator

Mutant operators: Examples Mutant operator In P In mutant Variable replacement z=x*y+1; x=x*y+1; z=x*x+1; Relational operator replacement if (x<y) if(x>y) if(x<=y) Off-by-1 z=x*y+1; z=x*(y+1)+1; z=(x+1)*y+1; Replacement by 0 z=x*y+1; z=0*y+1; z=0; Arithmetic operator replacement z=x*y+1; z=x*y-1; z=x+y-1;

What is program mutation? § P’ is known as a mutant of P. § There might be a test t in T such that P(t)≠P’(t). In this case we say that t distinguishes P’ from P. Or, that t has killed P’. § There might be not be any test t in T such that P(t)≠P’(t). In this case we say that T is unable to distinguish P and P’. Hence P’ is considered live in the test process.

Killing Mutants Given a mutant m M for a ground string program P and a test t, t is said to kill m if and only if the output of t on P is different from the output of t on m. • Testers can keep adding tests until all mutants have been killed – Dead mutant : A test case has killed it – Stillborn mutant : Syntactically illegal – Trivial mutant : Almost every test can kill it – Equivalent mutant : No test can kill it (equivalent to original program)

Error detection using mutation § Consider the following function foo that is required to return the sum of two integers x and y. int foo(int x, y){ return (x-y); }

Error detection using mutation § Now suppose that foo has been tested using a test set T that contains two tests: T={ t 1: <x=1, y=0>, t 2: <x=-1, y=0>} § First note that foo behaves perfectly fine on each test in, i. e. foo returns the expected value for each test case in T. Also, T is adequate with respect to all control and data flow based test adequacy criteria.

Error detection using mutation Let us evaluate the adequacy of T using mutation. Suppose that the following three mutants are generated from foo. M 1: int foo(int x, y){ return (x+y); } M 2: int foo(int x, y){ return (x-0); } M 3: int foo(int x, y){ return (0+y); } § Note that M 1 is obtained by replacing the - operator by a + operator, M 2 by replacing y by 0, and M 3 by replacing x by 0.

Error detection using mutation Next we execute each mutant against tests in T until the mutant is distinguished or we have exhausted all tests. Here is what we get. T={ t 1: <x=1, y=0>, t 2: <x=-1, y=0>} Test (t) foo(t) M 1(t) M 2(t) M 3(t) t 1 t 2 1 -1 0 0 Live Killed 1 -1

Questions About Mutation • Should more than one operator be applied at the same time ? –Should a mutated string contain one mutated element or several? –Almost certainly not – multiple mutations can interfere with each other –Extensive experience with programbased mutation indicates not

Mutation testing Consider the following program P 1. main(argc, argv) 2. int argc, r, i; 3. char *argv[]; 4. { r = 1; 5. for i = 2 to 3 do 6. if (atoi(argv[i]) > atoi(argv[r])) r = i; 7. printf(“Value of the rank is %d n”, r); 8. exit(0); } 16

Mutation testing • Test Case 1: input: 1 2 3 output: Value of the rank is 3 • Test Case 2: input: 1 2 1 output: Values of the rank is 2 • Test Case 3: input: 3 1 2 output: Value of the rank is 1 17

Mutation testing 18 Mutant 1: Change line 5 to for i = 1 to 3 do Mutant 2: Change line 6 to if (i > atoi(argv[r])) r = i; Mutant 3: Change line 6 to if (atoi(argv[i]) >= atoi(argv[r])) r = i; Mutant 4: Change line 6 to if (atoi(argv[r]) > atoi(argv[r])) r = i; Execute modified programs against the test suite, you will get the results: Mutants 1 & 3: Programs will pass the test suite, i. e. , mutants 1 & 3 are not killable Mutant 2: Program will fail test cases 2 Mutant 1: Program will fail test case 1 and test cases 2 Mutation score is 50%, assuming mutants 1 & 3 nonequivalent

Logic Mutation Testing Process 1) Systematically create program variations (mutants) each containing a single typical logic fault if (a && b) if (a || b) 2) For each mutant, find an input that distinguishes the original program from the mutant program (killing) - a && b and a || b evaluate to different truth values

Program-based Grammars Original Method int Min (int A, int B) { int min. Val; min. Val = A; if (B < A) { min. Val = B; } return (min. Val); } // end Min

With Embedded Mutants int Min (int A, int B) { int min. Val; min. Val = A; ∆ 1 min. Val = B; if (B < A) ∆ 2 if (B > A) ∆ 3 if (B < min. Val) { min. Val = B; ∆4 Bomb (); ∆5 min. Val = A; ∆6 min. Val = fail. On. Zero (B); } return (min. Val); } // end Min Replace one variable with another Changes operator Immediate runtime failure … if reached Immediate runtime failure if B==0 else does nothing

Syntax-Based Coverage Criteria 1) Strongly Killing Mutants: Given a mutant m M for a program P and a test t, t is said to strongly kill m if and only if the output of t on P is different from the output of t on m 2) Weakly Killing Mutants: Given a mutant m M that modifies a location l in a program P, and a test t, t is said to weakly kill m if and only if the state of the execution of P on t is different from the state of the execution of m immediately on t after l

Weak Mutation Example • Mutant 1 in the Min( ) example is: min. Val = A; ∆ 1 min. Val = B; if (B < A) min. Val = B; • The complete test specification to kill mutant 1: • (A = 5, B = 3) will weakly kill mutant 1, but not strongly

Equivalent Mutation Example • Mutant 3 in the Min() example is equivalent: min. Val = A; if (B < A) ∆ 3 if (B < min. Val) • The infection condition is “(B < A) != (B < min. Val)” • However, the previous statement was “min. Val = A” – Substituing, we get: “(B < A) != (B < A)” • Thus no input can kill this mutant

Strong Versus Weak Mutation (X = -6) will kill mutant 4 under weak mutation 1 boolean is. Even (int X) 2 { 3 if (X < 0) 4 X = 0 - X; ∆4 X = 0; 5 if (float) (X/2) == ((float) X) / 2. 0 6 return (true); 7 else 8 return (false); 9 }

Testing Programs with Mutation Prog Input test method Define threshold Create mutants Run equivalence detector Automated steps no Threshold reached ? Fix P no P (T) correct ? Generate test cases Run T on P Run mutants: • schema-based • weak • selective Eliminate ineffective TCs yes 26

Mutation Operators for Java 1. ABS –– Absolute Value Insertion: Each arithmetic expression (and subexpression) is modified by the functions abs(), neg. Abs(), and fail. On. Zero(). 2. AOR –– Arithmetic Operator Replacement: Each occurrence of one of the arithmetic operators +, －, *, ／, and % is replaced by each of the other operators. In addition, each is replaced by the special mutation operators left. Op, and right. Op. 3. ROR –– Relational Operator Replacement: Each occurrence of one of the relational operators (<, ≤, >, ≥, =, ≠) is replaced by each of the other operators and by false. Op and true. Op.

Mutation Operators for Java (2) 4. COR –– Conditional Operator Replacement: Each occurrence of one of the logical operators (and - &&, or - || , and with no conditional evaluation - &, or with no conditional evaluation - |, not equivalent - ^) is replaced by each of the other operators; in addition, each is replaced by false. Op, true. Op, left. Op, and right. Op. 5. SOR –– Shift Operator Replacement: Each occurrence of one of the shift operators <<, >>, and >>> is replaced by each of the other operators. In addition, each is replaced by the special mutation operator left. Op. 6. LOR –– Logical Operator Replacement: Each occurrence of one of the logical operators (bitwise and - &, bitwise or - |, exclusive or - ^) is replaced by each of the other operators; in addition, each is replaced by left. Op and right. Op.

Mutation Operators for Java (3) 7. ASR –– Assignment Operator Replacement: Each occurrence of one of the assignment operators (+=, -=, *=, /=, %=, &=, |=, ^=, <<=, >>>=) is replaced by each of the other operators. 8. UOI –– Unary Operator Insertion: Each unary operator (arithmetic +, arithmetic -, conditional !, logical ~) is inserted in front of each expression of the correct type. 9. UOD –– Unary Operator Deletion: Each unary operator (arithmetic +, arithmetic -, conditional !, logical~) is deleted.

Mutation Operators for Java (4) 10. SVR –– Scalar Variable Replacement: Each variable reference is replaced by every other variable of the appropriate type that is declared in the current scope.

Binary search example int search ( int key, int [] elem. Array){ int bottom = 0; int top = elem. Array. length - 1; int mid; int result = -1; while ( bottom <= top ) { mid = (top + bottom) / 2; if (elem. Array [mid] == key){ result = mid; return result; } // if part else { if (elem. Array [mid] < key) bottom = mid + 1; else top = mid - 1; } } //while loop return result; }