Something to amuse you http xkcd com518 http

Something to amuse you… http: //xkcd. com/518 http: //www. youtube. com/watch? v=k 0 xgj. Uh. EG 3 U 70 minutes CS 2212 - UWO 1

Find the problem (i. e. Is line/statement coverage enough? ): • This code prints out 1 if the number is even and 1 if the number is positive and 0 for odd and 0 for negative. • I tested it with a minimal test case (x=6) to get 100% line coverage (see image), and I got the correct result but still the program is wrong. WHY? ? ? include <stdio> main () { int x, pos, even; printf("Enter an integer: "); scanf("%d", &x); pos = 1; even = 1; if (x > 0) {pos = 1; } if (x%2 == 0) {even = 1; } printf (“Number: %d, Positive: %d, Even: %dn", x, pos, even); return 0; } CS 2212 - UWO 2

Structural Testing Structural testing measures • Flowcharts • Structural testing and flowcharts • Minimal test suites CS 2212 - UWO 3

Program Flowcharts Example of code: main () { int x, pos, even; printf("Enter an integer: "); scanf("%d", &x); pos = 0; even = 0; if (x > 0) {pos = 1; } if (x%2 == 0) {even = 1; } printf ("%d, %dn", x, pos, even); return 0; } • Intended to set pos = 1 iff x is positive, even = 1 iff x is even Draw a graph of the statements and decisions in the code: • Statements = boxes • Decisions = diamonds This graph called a “flowchart” CS 2212 - UWO 4

Flowchart Example Printf() Scanf() T X>0 F T X%2=0 Pos=0 Even=0 Pos=1 Even = 1 F Printf() End CS 2212 - UWO 5

Flowchart Example A flowchart is really a labeled directed graph • Boxes and diamonds = nodes • Arrows = edges Edges indicate parts of “paths” which may be followed through code Labels on edges going out of diamonds indicate what happens when decision in diamond is true/false CS 2212 - UWO 6

Flowcharts: Example 2 (Code) i = 0; while (s[i] != 0) { if (s[i] >= ‘a’) && (s[i] <= ‘z’)) { s[i]=s[i] – 32; } i++; } CS 2212 - UWO 7

Flowcharts: Example 2 (Flowchart) i=0 F s[i]!=0? T F s[i]>=‘a’&& s[i]<=‘z’? T s[i]=s[i]-32 i++ End CS 2212 - UWO 8

Structural Testing and Flowcharts What does this have to do with testing? If test suite executes all statements, then test suite visits every box and diamond in the flowchart • We say the test suite covers every node • We can also say the test suite covers every statement This does not mean that every arrow (edge) in the flowchart has been followed Example: consider the first example flowchart • We run it on one test case: x=2 • All nodes visited • However, neither F branch has ever been taken If every arrow in the flowchart has been taken by some case in the test suite: • We say the test suite covers every edge • We can also say the test suite covers every decision CS 2212 - UWO 9

Edge vs. Node Coverage Edge cover is more thorough than node coverage • Catches more problems Example: consider first example flowchart • Change statements pos =0; even = 0 in the first column to pos = 1; even = 1 • Now, program always sets pos and even to 1 • But we wanted pos to indicate whether x is positive, even to indicate whether x is even • If we just test with x = 2 CS 2212 - UWO 10

Would 100% Statement/Node Coverage Find the Mistake with test case x=2? Printf() Coding mistake made by a novice programmer (initialized incorrectly) Scanf() T x>0 F T x%2=0 Pos=1 Even=1 Pos=1 Even = 1 F Printf() End CS 2212 - UWO 11

• If we test with both x = 2 and x = -1: —We cover every edge —We also force a failure to happen, program reports wrong results for x = - 1 Thus, edge coverage can reveal failures where node coverage cannot. CS 2212 - UWO 12

Coverage Terminology If a test suite executes 100% of statements in program, we say it achieves • 100% node coverage or 100% statement coverage If a test suite follows 90% of edges in flowchart, we say it achieves 90% edge coverage (etc) “Line Coverage”: what gcov measures • Any line containing code is measured —Considered covered if any code on line is executed • Not exactly the same thing as statement coverage —Can have more than one statement on a line • However, 100% statement coverage implies 100% line coverage Edge coverage also called decision coverage • Tests each decision in an if, while, etc. both ways CS 2212 - UWO 13

Strength of Coverage Measures 100% node coverage implies 100% line coverage • However, 100% line coverage does not imply 100% node coverage 100% edge coverage implies 100% node coverage • If followed every edge, must have visited every node • However, as we saw, 100% node coverage does not necessarily imply 100% edge coverage Edge (Decision) Thus we say: • Node (statement) coverage is strong than line coverage • Edge (decision) coverage is stronger than node coverage We can draw a graph: • Arrow from A to B: B is a stronger coverage measure than A CS 2212 - UWO Node (Statement) Line 14

Question Draw a flowchart for the following c code and find the minimum number of test cases to achieve: int main () • 100% line coverage { • 100% node coverage int a, b; a = 6; • 100% edge coverage printf("nn. Enter a value for a: "); scanf ("%d", &a); printf("nn. Enter a value for b: "); scanf ("%d", &b); if (( a<10) || (a > 15)) if (b ==9) printf ("goodn"); else printf("eveningn"); else printf("madam"); printf("sirn"); return 0; } CS 2212 - UWO 15

Your Answer CS 2212 - UWO 16

Minimal Test Suites By throwing more and more test cases into a test suite, we may achieve higher coverage However, longer test suites take longer to run Hence, we often want to find some minimal test suite • Important to define what minimal means i. e. minimal test cases? Minimal number of input used? Minimum time to run? Example: first flowchart example: • Test suite: x=2, x=1, x=-2 covers all edges • However, it has 3 test cases • Test suite: x=2, x=-1 covers al edges and has only 2 test cases • Cannot have a minimal test suite which covers all edges and has only 1 test case —Have to execute each decision both ways Hence, x=2, x=-1 is a minimal test suite for 100% edge coverage • Minimal in the sense of needing the fewest test cases CS 2212 - UWO 17

Coverage and Test Suites: Another Example Consider again the case-conversion code flowchart (Example 2) Find “minimal” test suite which achieves: • 100 % node coverage • 100% edge coverage Decide what minimal means in this context! CS 2212 - UWO 18

Case Conversion Flowchart: Coverage Clearly we can achieve 100% node coverage with one-char test case • Say, for example: “a” This test case also follows both arrows from the top diamond • This is because: —First time through loop (i==0), T branch taken —Next time condition is evaluated (i==1), s[i] is the null (end of string) character —Hence, F branch is also taken However, F branch of other diamond not taken To cover all edges, we need test suite like • (A) s= “a”, s=“X”; or • (B) s = “a. X” CS 2212 - UWO 19

Case Conversion Flowchart: Minimality Which of (A) and (B) is “minimal” • Depends on how we define minimality • (A) has 2 test cases, but each has just 1 character • (B) has just 1 test case, but it has 2 characters We should always define precisely what we mean by minimality for a given test suite • Here, makes more sense to accept (B) as minimal • Don’t’ have to run program as many times, incur overhead of starting up process again Hence, for this problem, we should say something like: • Test suite X will be considered smaller than test suite Y if either: —X has few test cases than Y; or —X has the same number of test cases as Y, but the total number of characters in X is smaller • Saves us from having to accept s = “aaaaaa. XXXXXX” as minimal CS 2212 - UWO 20

Stronger Coverage Measures So far we have been writing compound decisions • (e. g. (c >=‘a’) && (c <=‘z’)) inside a single diamond Decision: the whole thing in parentheses after the “if”, while”, etc • Consider the code: if ((c>=‘a’) && (c<=‘z’)) {. . } • (c >=‘a’) && (c<=‘z’) is the decision Conditions: the individual terms of the condition, connected by logical operators • (c>=‘a’) && (c<=‘z’) are the two conditions If we separate each condition with the decision into a separate diamond: • We can get a better reflection of what the program does CS 2212 - UWO 21

Splitting Up Diamonds: Flowcharts Example: • Instead of writing F we write s[i]>=‘a’ && s[i]<=‘z’ T s[i]>=‘a’ F T s[i]<=‘z’ F T • Now we have two new arrows • To cover all arrows, we have to cover the two ones as well CS 2212 - UWO 22

Short Circuit Condition Coverage If we split up all diamonds and then cover all arrows: • We are achieving more thorough coverage than just edge (decision) coverage can give us This is called short circuit (SC) condition coverage • Recall definition of short circuit evaluation —Evaluation of a Boolean expression is cut short as soon as it can be determined to be true or false • Applies to language with short circuit evaluation of decisions, e. g. C, C++, Java Similar but not identical to another notion called just “condition coverage” • Condition coverage applies to languages without short circuit evaluation of decisions, e. g. Visual Basic • We will not study this coverage measure CS 2212 - UWO 23

Achieving SC Condition Coverage Consider the case conversion example Test case “a. X” is sufficient to achieve 100% edge coverage • Whole decision (c>=‘a’) && (c<=‘z’) is tested both ways: true for ‘a’ and false for ‘X’ Is not sufficient to achieve 100% SC condition coverage • Have not take the F arrow from the s[i] <= ‘z’ diamond To take that arrow we need a character that is: • Greater than or equal to ‘a’ • Greater than ‘z’ We need one such ASCII character • The test case “a. X~” will achieve 100% SC condition coverage CS 2212 - UWO 24

Achieving SC Condition Coverage in General In general, for a decision A && B to achieve SC condition coverage, we need to design test cases to make: • A true, B true • A true, B false • A false We will not even be able to get to the evaluation of B unless A is true as in first two cases (due to SC evaluation) Similarly, for a decision A || B to achieve SC condition coverage, we need to design test cases to make: • A false, B true • A false, B false • A true We will not even be able to get to the evaluation of B unless A is false as in the first two cases CS 2212 - UWO 25

SC Condition Coverage vs. Edge Coverage If we have achieved SC condition coverage: • We must have evaluated each condition of an “if”, “while”, etc. both ways • Therefore, we must have evaluated each decision both ways Therefore, SC condition coverage is stronger than edge (decision) coverage Is there anything stronger than SC condition coverage? CS 2212 - UWO 26

Path Coverage “ 100% path coverage”: every possible path through the flowchart of the code has been executed by some test case • Path: sequence of nodes visited during an execution —Can contain repeated elements if execution has visited a node more than once • The strongest possible coverage measure For code with no loops, this is achievable For code with while loops: • Going around loop a different number of times means we have taken a different path For some code, we can go around a loop any number of times Therefore, for some code: • it is not possible to achieve 100% path coverage CS 2212 - UWO 27

Path Coverage: Example 1 Integer classification example again: Four paths through the code are possible • X=2 • X=1 • X = -2 • X = -1 Draw the paths for each of the above examples! Therefore we need four test cases for 100% coverage Printf() Scanf() X>0 T F T X%2=0 Pos=0 Even=0 Pos=1 Even = 1 F Printf() End CS 2212 - UWO 28

Path Coverage: Example 2 i=0 Case conversion example again Every string of a different length will follow a different path F s[i]!=0? T F s[i]>=‘a’&& s[i]<=‘z’? T s[i]=s[i]-32 i++ End CS 2212 - UWO 29

Path Coverage for Loops Path coverage for many while loops is impossible Most programs have while loops! Most have at least one that can be followed an arbitrary number of times: • e. g. A top level loop where we read a command execute it Therefore, for most whole programs, we don’t even attempt 100% path coverage Path coverage is still useful for small sections of code CS 2212 - UWO 30

Strength of Coverage Measures Path (strongest) Edge (Decision) Node (Statement) Line (weakest) Where arrow from A to B means “B is stronger than A” CS 2212 - UWO 31

Testing Loops If we have loops, we can’t test all possible paths However, just decision coverage may not spot usual kinds of errors Often code is wrong because programmer didn’t consider what would happen when: • Loop decision is false right from the start • Loop decision is true once, then false Sometimes there is a maximum number (max) of times loop can be executed • Example: loop may stop at end of an array, or when some condition is true In these cases, code may be wrong because programmer didn’t consider what would happen when • Loop decision is true max times • Loop decision is true max-1 times CS 2212 - UWO 32

Loop Coverage It is therefore useful to write test cases which: • Execute loop 0 times • Execute loop 1 time • Execute loop more than 1 time • Execute loop max times (if max exists) • Execute loop max -1 times (if max exists) These tests are often referred to as loop coverage • However, there is no real formal definition of loop coverage as with node, edge etc. coverage CS 2212 - UWO 33

Loop Coverage: Exercise Provide adequate loop coverage for the following: (code finds the rightmost dot in a string, used to find extension for filename) i = strlen(fname) -1; while (i>0 && fname[i] != ‘. ’) {i--; } To do “loop coverage”, list some of the possible test cases: fname contains: “report 1. ” • (0 times) fname contains: “report 1. t” • (1 time) fname contains: “report 1. txt” • (more than 1 time) fname contains: “. txt” • (max – 1 times) fname contains: “report 1” • (max times) Note: max here is the length of the string minus 1 Note: the code above is almost certainly correct • However, problem may not be with the loop itself, but the code that comes after • For instance, maybe the code that comes afterward: —Expects a non-empty extension —Expects a non-empty filename before the extension —Can’t handle the case of no extension • Loop testing will help find those problems too CS 2212 - UWO 34