Software Testing Sudipto Ghosh CS 406 Fall 99

Software Testing Sudipto Ghosh CS 406 Fall 99 November 9, 1999 11/09/99 CS 406 Testing

Learning objectives • Functional testing • Equivalence class partitioning • Boundary value analysis • Structural testing • Control flow-based 11/09/99 CS 406 Testing 2

Testing for correctness • Identify the input domain of P. • Execute P against each element of the input domain. • For each execution of P, check if P generates the correct output as per its specification S. 11/09/99 CS 406 Testing 3

Input domain • The set of all valid inputs that a program P can expect is the input domain of P. • The size of an input domain is the number of elements in it. • An input domain could be finite or infinite. • Finite input domains may be large! 11/09/99 CS 406 Testing 4

Identifying input domains • For the sort program: • N: size of the sequence, N < 3 • Sequence of N integers, such that for any integer K, 0 K (e – 1), e = 3 • K: each element in the sequence • Some sequences are: • [ ] : An empty sequence (N = 0) • [0] : A sequence of size 1 (N = 1) • [2 1] : A sequence of size 2 (N = 2) 11/09/99 CS 406 Testing 5

Size of input domain • Suppose that 0 N 106 • The size of the input domain is the number of all sequences of size 0, 1, 2… • The size is computed as: • To test for correctness, P needs to be tested on all inputs. • How many years would that take? 11/09/99 CS 406 Testing 6

Exhaustive testing • Previous example • P is executed on all elements in the input domain. • For most programs, exhaustive testing is not feasible • What is the alternative? 11/09/99 CS 406 Testing 7

Partition testing • Input domain is partitioned into a finite number of sub-domains. • P is then executed on a few elements in each sub-domain. • Four sub-domains for our N=0 1 example: N=2 N=3 [], [2 0], [2 1 0] 3 4 N=1 Reduced from 85 to 4 ! 2 11/09/99 CS 406 Testing 8

Confidence in your program • Confidence is a measure of one’s belief in the correctness of the program • Correctness is not measured in binary terms: a correct or incorrect program • Measured as a probability of correct operation of a program when used in various scenarios • Reliability • Test completeness: Extent to which a program has been tested and errors have been found and removed 11/09/99 CS 406 Testing 9

Types of module testing • “Informal testing” performed by programmer while developing module • Methodical testing performed by SQA group • Nonexecution-based testing • Execution-based testing • Haphazard data as input • Systematically constructed test cases • Testing to specifications • Testing to code 11/09/99 CS 406 Testing 10

Testing to specifications • • • Black-box testing Data-driven testing Input/output-driven testing Functional testing Code is ignored, i. e. the internal structure of the code is not important • The specification document is the only information used for creating test cases 11/09/99 CS 406 Testing 11

Testing to code • • • White-box testing Glass-box testing Logic-driven testing Path-oriented testing Specifications are not used to generate the test cases • Test “to code, ” i. e. the internal structure of the code is important 11/09/99 CS 406 Testing 12

What is functional testing? • Functional testing tests how well a program meets the functionality requirements. • When test inputs are generated using program specifications, we say that we are doing functional testing. 11/09/99 CS 406 Testing 13

Feasibility of functional testing • Problem with exhaustive testing • Very large (possibly infinite) number of test cases. • Need to devise: • Small, manageable set of test cases. • Maximize the chances of detecting faults. • Minimize the chances of wasting test cases by having more than one test case detecting the same fault. 11/09/99 CS 406 Testing 14

Equivalence class partitioning • Divide the domain of all inputs into a set of equivalence classes. • If any test in an equivalence class succeeds, then every test in that class will succeed. • How would you get ideal equivalence classes? • Difficult without looking at the internal structure • Difficult even with the internal structure 11/09/99 CS 406 Testing 15

Equivalence class partitioning Set of all inputs Specifications Put those inputs together for which the behavior pattern of the module is specified to be different into similar groups Equivalence classes 11/09/99 CS 406 Testing 16

Rationale behind equivalence class partitioning We assume that if the specifications require exactly the same behavior from each element in a class of values, then the program is likely to be constructed so that it either succeeds or fails for each of the values in that class. 11/09/99 CS 406 Testing 17

Guidelines for partitioning • For robust software, we must test for incorrect inputs too. • For each equivalence class of valid inputs, we have equivalence classes of invalid inputs. • Input condition specifies a range a X b. • a X b (Valid case) • X < a, and X > b (invalid cases) • Input specifies a value • create one for the valid value • create two for incorrect (one above, one below) 11/09/99 CS 406 Testing 18

Guidelines for partitioning • Input specifies a value (contd. ) • For boolean, only one value • Input condition specifies a member of a set • Create one for the valid value • Create one for invalid value (not in the set) • Example: • Factorial (n), where n 0 • Valid classes {0}, {x | x 1} • Invalid class {x | x < 0} 11/09/99 CS 406 Testing 19

Non-overlapping partitions • In the previous example, the equivalence classes were non-overlapping, i. e. , the sub-domains were disjoint. • It is sufficient to pick one test from each equivalence class to test the program. • An equivalence class is considered covered if at least one test has been selected from it. • Our goal is to cover all equivalence classes. 11/09/99 CS 406 Testing 20

Overlapping partitions • Suppose a program P takes three integers, X, Y and Z. It is known that: • X<Y • Z>Y 11/09/99 X<Y Z>Y X Y Z>Y CS 406 Testing 21

Overlapping partition: test selection • Select 4 test sets as • • X=4, Y=7, Z=1 (satisfies X < Y) X=4, Y=2, Z=1 (satisfies X Y) X=1, Y=7, Z=9 (satisfies Z > Y) X=1, Y=7, Z=2 (satisfies Z Y) • Can also reduce the number of test cases to 2 • X=4, Y=7, Z=1 (satisfies X < Y and Z Y) • X=4, Y=2, Z=3 (satisfies X Y and Z > Y) 11/09/99 CS 406 Testing 22

Boundary Value Analysis (BVA) • Observation: • Programs that work correctly for a set of values in an equivalence class, fail on some special values. • These values lie on the boundary of the equivalence class. • Choose an input from a test case from an equivalence class such that the input lies at the edge of the equivalence class. • These test cases are “extreme cases” 11/09/99 CS 406 Testing 23

BVA examples • Suppose the range is 0. 0 X 1. 0. • • 0. 0 (valid input) 1. 0 (valid input) -0. 1 (invalid input) 1. 1 (invalid input) • For a list • first and last element of the list • A program takes a string S and integer X as input such that • a X b, and length(S) 100. Derive tests. 11/09/99 CS 406 Testing 24

BVA analysis • A program takes two integers X and Y, such that a X b, c Y d. • How many test cases do we get? d c a 11/09/99 b CS 406 Testing 25

Output variables • Equivalence class partitioning can be applied to output variables • BVA also can be applied to output data 11/09/99 CS 406 Testing 26

Testing functions • Identify each function implemented in a module • Devise test data to test each function separately • Module may contain a hierarchy of lower level functions connected by program control structures • Perform functional testing recursively • Higher level: black-box • Lower level: glass-box 11/09/99 CS 406 Testing 27

Problems with previous approach • Usually higher level functions are not as structured • Lower level functions are intertwined • Functionality does not coincide with module boundaries • Distinction between module testing and integration testing is blurred • Testing one module is not possible without simultaneously testing other modules whose functionality is used 11/09/99 CS 406 Testing 28

Structural testing • Intent is to exercise the different programming structures and data structures used in the program • Intent is not to exercise all the different input and output conditions • though this may be a by-product • Achieve test cases that force the desired “coverage” of different structures • Criteria are formal and precise 11/09/99 CS 406 Testing 29

Control-flow based criteria • Statement coverage • Run a series of test cases and ensure that every statement is executed at least once • Simplest form of glass box testing • What is the weakness? Consider the example: int abs (int x) { if (x >= 0) Test inputs: Error x = 0 - x; x = 5: What is coverage? return x; x = 0: What is coverage? } 11/09/99 CS 406 Testing 30

Statement coverage • Not very strong, may leave errors undetected. • Examples: • if statement without else part • conjunctions of predicates • In all these cases, all branches were probably not exercised. • Can you think of a better criterion based on the above observation? 11/09/99 CS 406 Testing 31

Branch coverage • Require that every decision is evaluated to true and false values at least once during testing • Branch coverage implies statement coverage • Each statement is part of some branch 11/09/99 CS 406 Testing 32

Control flow graph of a program • Let G be the graph of a program P. • Node: • Represents a block of statements that are always executed together • Edge (i, j) from node i to node j: • Represents a possible transfer of control after executing the last statement of the block represented by node i to the first statement in the block represented by node j. 11/09/99 CS 406 Testing 33

Control flow graph of a program • Start node: • Node corresponding to a block whose first statement is the start statement of P. • Exit node: • Node corresponding to a block whose last statement is an exit statement of P. • Path: • Finite sequence of nodes (n 1, n 2, …, nk), k>1 such that there is an edge (ni, ni+1) for all nodes ni in the sequence, except the last node nk. 11/09/99 CS 406 Testing 34

Control flow graph of a program • Complete Path: • Path whose first node is a start node and the last node is an exit node. • All-nodes criterion (statement coverage) • All-edges criterion (branch coverage) 11/09/99 CS 406 Testing 35

An example (xy) 1. 2. 3. 4. 5. 6. 7. 8. 9. scanf(x, y); if(y < 0) pow = 0 – y; else pow = y; z = 1. 0; while(pow != 0) { z = z * x; pow = pow – 1; } if ( y < 0 ) z = 1. 0/z; printf(z); 11/09/99 CS 406 Testing 36

Control Flow Graph of example 1 3 2 4 5 6 7 8 11/09/99 CS 406 Testing 9 37

Problems with branch coverage • What if a decision has many conditions (using and, or) • Decision may evaluate to true or false without actually exercising all the conditions Test inputs: int check (int x) { x = 5: if ((x >= 5) && (x <= 200)) x = -5: return TRUE; Error return FALSE; (should be 100) } 11/09/99 CS 406 Testing 38

Solution? • Require all individual conditions to evaluate to true and false • Problem: • Even if individual conditions evaluate to true and false, the decision may not get both true and false values • Solution: • Require both decision / condition coverage!! • Still there is a problem. 11/09/99 CS 406 Testing 39

Path testing • Some errors are related to some combinations of branches. • Presence revealed by an execution of a path that includes those branches. • Solution: • Require all possible paths in the CFG to be executed during testing. • Path-coverage criterion, or all-paths criterion • Path coverage implies branch coverage 11/09/99 CS 406 Testing 40