WhiteBox Testing Using Pex Snigdha Athaiya Formal Methods

White-Box Testing Using Pex Snigdha Athaiya Formal Methods in Software Engineering *Slides based on the material on Pex by K. V. Raghavan and presentation by Tao Xie titled “Parameterized Unit Testing using Pex”

What is a Unit Test? A unit test is a small program with assertions, scrutinizing the smallest testable parts of an application. Annotation Selecting test inputs Stating appropriate assertions [Test. Method] public void Add. Test() { Hash. Set set = new Hash. Set(); set. Add(3); set. Add(14); Assert. Are. Equal(set. Count, 2); } Meaningful Sequence of method calls

Unit Tests • smallest testable parts or units can be procedures, classes or even logical modules • Purpose of unit testing – to isolate sections of a program and ensure their correctness • Correctness? It is with respect to the specifications of the unit under test. • If source code is changed due to addition of specifications, then a new set of test cases are written. The new code should still pass the old tests. A growing body of test cases is always desirable.

Writing a Unit test - Demo

Unit Testing: Problems • Quality of unit tests • Do they test what is important? • Amount of unit tests • How many tests to write?

Unit Testing: Measuring Quality • Coverage: Are all parts of the program exercised? • • statements basic blocks explicit/implicit branches … • Assertions: Does the program do the right thing? • Just high coverage or large number of assertions is not a good quality indicator. • Only both together are!

Unit Testing: Measuring Quality • Coverage: Are all parts of the program exercised? • • statements basic blocks explicit/implicit branches … • Assertions: Does the program do the right thing? • We will focus on the first part for now, i. e. coverage

The Testing Problem Given, a sequential program P, with conditionals C, compute a set of program inputs I such that : for all conditionals c in C and for each branch b of c, if b is reachable in some execution of P, then there exists an input i in I such that the execution of P on input i goes through branch b.

Pex • test input generation framework • generates inputs to achieve maximum branch coverage • known C# exceptions treated as implicit conditionals • analyzes the execution paths • at the level of. NET instructions • Incremental analysis that discovers feasible execution paths • represents execution paths as constraints and uses the constraint solver Z 3 for checking feasibility • uses Dynamic Symbolic Execution • a combination of static and dynamic analysis

Beginning execution with symbols for x with x=0 and y=1 and y x = 1, y = 0 x = 0, y = 1 Pex – Path Exploration m(x, y) { b 1: if (x > 0) t = 1 else t = 0 b 2: if (y > 0) x = x + t else x = x - t b 3: if (t > 0) z = x else z = x + 1 } F z = 1 t = 1 x = 0 b 3 b 2 T t = 0 b 2 b 1 x = 0 b 3 z = 0 z = 1

Pex Algorithm

Key Terminology F b 1 T • Symbolic Expression – an expression involving the initial symbolic values of variables. b 2 • Symbolic State – mapping from variables to Symbolic Expressions. b 3

Key Terminology • Static Conditional is a conditional node in the program. Each of the two branches out of a static conditional is called a Static Branch. • A Dynamic Conditional is a conditional node in the execution tree, and is an instance of a static conditional. Each of the two branches out of a dynamic conditional is called a Dynamic Branch b 3 F b 1 T m(x, y) { b 1: if (x > 0) t = 1 else t = 0 b 2: if (y > 0) b 2 x = x + t else x = x - t b 3: ifb 3 (t > 0)b 3 z = x else z = x + 1 }

Key Terminology • A Branch Constraint is a Boolean expression annotating a dynamic branch • An Execution Tree represents a set of execution traces. Each path from the root to a leaf represents a full execution trace from beginning to end of the method under test. The tree is factored, that is, any common prefix of one or more traces occurs only once in the tree. F b 1 T b 2 b 3 b 3

Outline of Pex Algorithm 1. Maintain a tree T of execution traces already taken. Initially it is empty. 2. Run given method m using default inputs. Let t be the trace obtained. 3. while(not all static branches covered){ 1. Add t to the tree T. 2. Compute symbolic states after assignment statements in t and branch constraints along all branches in t. 3. If t reached some static branch that has not yet been covered, use the input values that were used to obtain t to emit a unit test. 4. new. Trace. Found = false

5. while(there exists a dynamic conditional b in the tree T such that • one of its branches has been executed so far • the other branch is not present in the tree • AND iterations limit of the inner loop not yet hit • AND new. Trace. Found == false) { 1. Flip the constraint on the conditional b and feed the conjunction of all branch constraints from the root of the tree up to and including the flipped conditional to the solver. This conjunction is called a path constraint. 2. If(solver finds a solution I) 1. new. Trace. Found = true 3. else 1. Mark untaken branch out of b with an ‘X’ } 4. If new. Trace. Found == true 1. then execute method m on input I and let t be the resulting trace 2. else break }

Branch Flipping strategies • In inner loop condition we need to pick a condition b to flip. What if there are multiple candidates? • Simple DFS and BFS are naïve possibilities. Why? • Pex uses multiple strategies, and gives each one of them a turn (e. g. roundrobin) • Strategy 0: • For each static conditional node in the program maintain a count of how many times its instances have been flipped. At each turn choose the available one with the smallest count. • Strategy 1: • pick the shallowest dynamic conditional (based on path depth) i. e. BFS • When it is strategy i’s turn, if there are many dymanic branches that are in a tie, it could use strategy j to break the tie.

5. while(there exists a dynamic conditional b in the tree T such that • • one the AND of its branches has been executed so far other branch is not present in the tree iterations limit of the inner loop not yet hit new. Trace. Found == false) { 1. Flip the constraint on the conditional b and feed the conjunction of all branch constraints from the root of the tree up to and including the flipped conditional to the solver. This conjunction is called a path constraint. 2. If(solver finds a solution I) 1. new. Trace. Found = true 3. else 1. Mark untaken branch out of b with an ‘X’ } 4. If new. Trace. Found == true 1. then execute method m on input I and let t be the resulting trace 2. else break }

• Start with Strategy 0 Exit with solution of single candidate found Try Strategy 0 on suitable. Cond next strategy to try will be 1 Else Tie Breaker!! Try strategy 1 to choose a single candidate out of from. I No more strategies to use exit tie-breaker loop

Illustration of Branch Flipping Strategies b 1: b 2: Flipping the shallowest conditional with the lowest count while (x<10) x++; assert(x==11); b 1 Strategy 0: For each static conditional node in the program maintain a count of how many times its instances have been flipped. At each turn choose the available one with the smallest count. Strategy 1: pick the shallowest dynamic conditional (based on path depth) i. e. BFS b 1 b 2 Flipping the cond with the lowest count (0) b 2 next. Strat = 1 current. Strat = 0 next. Strat = 0 current. Strat = 0 will be used by the next cond current. Strat = 1 tie breaking using 0 tie breaking using 1 finding call

Parametrized Unit Tests • A parameterized unit test is a small program that takes some inputs and states assumptions and assertions. Takes inputs [Data. Test. Method] public void test. Div(int m, int n) Invoking the method { under test with Int. Math. Functions. SP obj = new Int. Math. Functions. SP(); variables, and not try constants { Int. Math. Functions. SP. Div. Result res = obj. div(m, n); Generalized Assert. Is. True(res. quotient * n + res. remainder == m); Assert. Is. True(res. remainder >= 0 && res. remainder < n); assertions } catch(Int. Math. Functions. SP. Illegal. Division ill. Ex){Assert. Is. True(n == 0); } catch(Exception e) { Assert. Is. True(false); } }

Parameterized Unit Testing • It is supported by all popular testing frameworks – NUnit, MSTest, JUnit, etc. • MSTest uses the annotation [Data. Test. Method] for the PUT (parameterized unit test) and different inputs to the parameterized test can be passed as a list of [Data. Row(…one set of params to PUT. . )] annotations. (see file Div. Structure. PUT. cs in the project) • PUTs can serve as specifications due to the presence of generalized assertions, which specify functional correctness of a method under test (MUT). • They can be leveraged by (automatic) test input generators, like Pex.

Benefit of Using PUTs with Pex •