WhiteBox Testing FP Firstyear Project Course ITU Denmark

White-Box Testing ( “FÅP”: First-year Project Course, ITU, Denmark ) Claus Brabrand [ brabrand@itu. dk ] Claus Brabrand, ITU, Denmark WHITE-BOX TESTING Feb 26, 2008

Outline n Motivation n n What is Testing? n n Methodology: ”coverage testing” Automation n n Relation to other programming-related tasks? White-box Testing n n Why bother with testing? How to automate testing Exercises n Training for the exam Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [2] Feb 26, 2008

Learning & Exam Goals n ”Product”: n ”Oral Exam”: n Today, we’ll ”train for the exam”… : -) Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [3] Feb 26, 2008

Software Errors n Mobile Phones n ’ 00 -… n n Cruise Control System Model n ’ 86 (Grady Booch) n n Accellerated after car ignition car crashes Baggage Handling System n ’ 94 -’ 95 (at Denver Int’l Airport) n Claus Brabrand, ITU, Denmark Freeze and odd behaviors (really annoying)! $ 360, 000 USD WHITE-BOX TESTING [4] Feb 26, 2008

Software Errors (cont’d) zzz z n Train Control System n ’ 98 (Berlin) n n Mars Pathfinder n July ’ 97 n n Periodic resets. . . on mars! Win 95/98 w/ 3 rd-Party Device Drivers n late ’ 90 es n Claus Brabrand, ITU, Denmark Train cancellations Dysfunction (“blue screen of death”)! WHITE-BOX TESTING [5] Feb 26, 2008

Software Errors (cont’d 2) n Therac-25 Radiation Therapy n ’ 85 -’ 87 n n Patriot Missile Guidance System n ’ 91 (Gulf War 1. 0) n n Accumulating rounding errors deaths Ariane V n ’ 96 (one of the most expensive bugs, ever) n Claus Brabrand, ITU, Denmark Massive overdoses (6 deaths / amputations)! Conversion from 64 -bit float to 16 -bit signed int WHITE-BOX TESTING [6] Feb 26, 2008

…and what about? ! n Surgical Laser Control System n n Air Plane Control System n n Dysfunction (plane crash)! Nuclear Powerplant Control System n Claus Brabrand, ITU, Denmark Oops…! Core melt-down (“China-syndrome”)! WHITE-BOX TESTING [7] Feb 26, 2008

Outline n Motivation n n What is Testing? n n Methodology: ”coverage testing” Automation n n Relation to other programming-related tasks? White-box Testing n n Why bother with testing? How to automate testing Exercises n Training for the exam Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [8] Feb 26, 2008

Errors! n Syntactic errors: n n (different kinds) Mal-formed program: Semantic errors: int square(int x) { return x*x *** syntax error } ’; ’ expected at line 2 int square(int x) { return n*n; *** symbol error at line 2 } undefined variable ”n” Symbol errors n Type errors int square(float x) { n Other semantic errors: return x*x; error at line 2 (e. g. uninitialized vars) }*** type function returns float, n n Logical errors: n Compiler: ”no errors” Claus Brabrand, ITU, Denmark int square(int x) { return x+x; } no errors WHITE-BOX TESTING [9] not int found!!! Feb 26, 2008

Int’l standard for evaluation of: Software Quality ISO 9126 ed to? s o p p u s ’s do what it W S e h t s ended? Doe t in s a k r o Does it w ity l a n o i ct n u How mu ch time/ memory bandwid /s th/… doe s the SW pace/consume E ffic ien cy F Usability Reliability ISO How robust is the SW wrt. incorrect 9126 inputs, ’^C’, external netwk failures, . . . ? How easy is the SW to understand? …and use? Po rta bil ity How easy is it to transfer and ada pt SW to new envir onment / platform ? Claus Brabrand, ITU, Denmark ? ity l i b a i a M in a t n ify the SW? d o m to it is sy a e w o H And fix errors? WHITE-BOX TESTING [ 10 ] Feb 26, 2008

Testing vs. Debugging? n Testing vs. Debugging: Regarding: Functionality: Efficiency: Quality Assurance: (Functionality) Testing (Performance) Testing Diagnosis: Debugging Profiling Purpose: Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [ 11 ] Feb 26, 2008

Testing vs. Debugging (cont’d) Program: 0110 1021 SYSTEMATIC (confidence? !? ) Quality assurance: Re 0110 1011 Functionality Testing Is program ok? Fix problem (reprogram) Evaluate test results Diagnosis: Debugging Determine problem? Claus Brabrand, ITU, Denmark WHITE-BOX TESTING 0110 1011 Document test results (greater confidence!) [ 12 ] Feb 26, 2008

Performance Testing vs. Profiling Program: 0110 1021 SYSTEMATIC (confidence? !? ) Quality assurance: Re- Performance Testing Efficient enough? Improve program 0110 (reprogram) 1011 Evaluate test results Diagnosis: Profiling Determine problem? Claus Brabrand, ITU, Denmark WHITE-BOX TESTING 0110 1011 Document test results (greater confidence!) [ 13 ] Feb 26, 2008

Testing…: n ”Testing is easy” (e. g. , ”random experimentation”) n ”Testing well is not easy” § Requires SYSTEMATIC approach; test-case § production § evaluation § documentation n ”Testing can never prove error absence” (i. e. , testing is an ”incomplete process”) Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [ 14 ] Feb 26, 2008

Appropriate Test Sampling? n Representative? n Comprehensive? n Quality? n Quantity? n …? Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [ 15 ] Feb 26, 2008

White-box vs. Black-box Test n White-box Testing: n n n (aka. , ”structural testing”) (aka. , ”internal testing”) Test focus: n Black-box Testing: n n (aka. , ”behavioral testing”) (aka. , ”external testing”) Test focus: source code n specification (manual) n = in(); tt n = n/2; odd(n) ? ~ ff n = 3*n+1; program out(n); spec Complementary Approaches!!! Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [ 16 ] Feb 26, 2008

”Software Testing” (R. Patton) n Background reading: ”Software Testing”, Ron Patton, Sams Publishing, 2006 n Part II (pp. 53 – 123); ”Testing Fundamentals”: n Type: -testing Static (before runtime) ”Examining the Code” (chapter 6) ”Examining the Spec. ” (chapter 4) Dynamic (at runtime) ”Testing w/ X-ray Glasses” (chapter 7) ”Testing w/ Blinders On” (chapter 5) Time: Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [ 17 ] Feb 26, 2008

Outline n Motivation n n What is Testing? n n Methodology: ”coverage testing” Automation n n Relation to other programming-related tasks? White-box Testing n n Why bother with testing? How to automate testing Exercises n Training for the exam Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [ 18 ] Feb 26, 2008

Test Coverage? n Method coverage: n n Statement coverage: n n Does every statement run (at least once)? Branch coverage: n n Does every method run (at least once)? Does every branch run (at least once)? Path coverage: n Does every path run (at least once)? Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [ 19 ] Feb 26, 2008

Statement coverage n Branch coverage: n n Does every branch run (at least once)? -Box ”Branch Coverage Testing” is: Efficient (fast) ! n Effective (thorough) ! n n Claus Brabrand, ITU, Denmark Good for complicated program logic (esp. ”initialization errors”) WHITE-BOX TESTING [ 20 ] Feb 26, 2008

Control Structures n Control Structures: n n Statements (or Expr’s) that affect ”flow of control”: if-else: true [syntax]n if ( Exp ) Stm 1 else Stm 2 if: Exp false Stm [semantics] n The expression must be of type boolean; if it evaluates to true, the given statement is executed, otherwise not. Claus Brabrand, ITU, Denmark confluence true [syntax]n if ( Exp ) Stm false Stm 2 Stm 1 [semantics] n The expression must be of type boolean; if it evaluates to true, Statement-1 is executed, otherwise Statement-2 is executed. n Exp WHITE-BOX TESTING confluence [ 21 ] Feb 26, 2008

Control Structures (cont’d) n confluence while: [syntax] n while ( Exp ) Stm [semantics] n The expression must be of type boolean; if it evaluates to false, the given statement is skipped, otherwise it is executed and afterwards the expression is evaluated again. If it is still true, the statement is executed again. This is continued until the expression evaluates to false. n Exp true Stm Exp 1; for: [syntax]n for (Exp 1 ; Exp 2 ; Exp 3) Stm [semantics] n Equivalent to: confluence true Exp 2 false Stm { Exp 1; while ( Exp 2 ) { Stm } Claus Brabrand, ITU, Denmark false Exp 3; } WHITE-BOX TESTING Exp 3; [ 22 ] Feb 26, 2008

Stm/Branch Coverage Testing n if: n n if-else: n n TEST condition true and false while: n n TEST condition true and false TEST zero, one, more-than-one iterations in loop for: n TEST zero, one, more-than-one iterations in loop Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [ 23 ] Feb 26, 2008

Example 1 Choice points? public static void main ( String[] args ) { int mi, ma; if (args. length == 0) /* 1 if-else System. out. println("No numbers"); else { mi = ma = Integer. parse. Int(args[0]); for (int i=1; i < args. length; i++) { /* 2 for int obs = Integer. parse. Int(args[i]); if (obs > ma) /* 3 if-else ma = obs; else if (mi < obs) mi = obs; /* 4 if } System. out. println(”min=" + mi + ", " + "max=" + ma); }} Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [ 24 ] */ */ Feb 26, 2008

Control-Flow Graph n int mi, ma; CFG: true args. length 1 == 0 System. out. println ("No numbers"); false mi = ma = Integer. parse. Int(args[0]); int i=1; true i < args. length 2 false int obs = Integer. parse. Int(args[i]); obs 3> ma true ma = obs; true false mi <4 obs mi = obs; false i++; System. out. println(”min=" + mi + ", " + "max=" + ma); Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [ 25 ] Feb 26, 2008

Coverage Table n ”Coverage Table”: Choice 1 ife true false 2 for zero-times once more-than-once 3 ife true false 4 if true false Claus Brabrand, ITU, Denmark Input property Data set No numbers A At least one number B Exactly two numbers C At least three numbers E N > current max C N current max D N cur max & N > cur min E (3 rd num) N cur max & N cur min E (2 nd num) WHITE-BOX TESTING [ 26 ] Feb 26, 2008

Expectancy Table n ”Expectancy Table”: Data set A B C D E Input [17] [27, 29] [39, 37] [49, 47, 48] Expected output ”no numbers” ”min=17, max=17” ”min=27, max=29” ”min=37, max=39” ”min=47, max=49” = Actual output ”no numbers” ”min=17, max=17” ”min=27, max=29” ”min=39, max=39” ”min=49, max=49” Advice: (i. e. , zeroes)s. ) Advice: sing same numbers in tests ’s 0 d te c e p x e reu ld memory. ) uage Avoid id o g o n e v s la A u y re n a s e m m in ti e e valu out som fault ITU, Claus Denmark (De. Brabrand, [ 27 ] Feb 26, 2008 ata lay. TESTING (DWHITE-BOX

Debugging ’ D ’ then reveals… public static void main ( String[] args ) { int mi, ma; if (args. length == 0) /* 1 if-else System. out. println("No numbers"); else { mi = ma = Integer. parse. Int(args[0]); for (int i=1; i < args. length; i++) { /* 2 for int obs = Integer. parse. Int(args[i]); if (obs > ma) /* 3 if-else ma = obs; else if (mi < obs) mi = obs; /* 4 if } n: hould have bee S System. out. println(”min=" + mi + ", " + (obs < mi) "max=" + ma); }} Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [ 28 ] */ */ Feb 26, 2008

Re-Test ! n …as debugging often introduces new errors ! Fixed Program: Coverage Table: Expectancy Table: Recall: no guarantee! Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [ 29 ] Feb 26, 2008

Example 2 public static void main ( String[] args ) { int mi 1 = 0, mi 2 = 0; if (args. length == 0) /* System. out. println("No numbers"); else { mi 1 = Integer. parse. Int(args[0]); if (args. length == 1) /* System. out. println("Smallest = " + mi 1); else { int obs = Integer. parse. Int(args[1]); if (obs < mi 1) { mi 2 = mi 1; mi 1 = obs; } for (int i = 2; i < args. length; i++) { obs = Integer. parse. Int(args[i]); if (obs < mi 1) /* { mi 2 = mi 1; mi 1 = obs; } else if (obs < mi 2) mi 2 = obs; } System. out. println("The two smallest are: " + mi 1 + " and " + mi 2); } } } Claus Brabrand, ITU, Denmark WHITE-BOX TESTING 1 if-else */ 2 if-else */ /* 3 if */ /* 4 for */ 5 if-else */ /* 6 if */ [ 30 ] Feb 26, 2008

Coverage Table (Ex. 2) Choice 1 ife true 1 ife false 2 ife true 2 ife false 3 if true 3 if false 4 for zero-times 4 for once 4 for more-than-once 5 ife true 5 ife false 6 if true 6 if false Claus Brabrand, ITU, Denmark Input property Data set A No numbers B At least one number B Exactly one number C At least two numbers C 2 nd number ≥ 1 st number D 2 nd number < 1 st number D Exactly two numbers E Exactly three numbers H At least four numbers E 3 rd number < current min F 3 rd number ≥ current min F 3 rd ≥ cur min & 3 rd < 2 nd least G 3 rd ≥ cur min & 3 rd ≥ 2 nd least WHITE-BOX TESTING [ 31 ] Feb 26, 2008

Expectancy Table (Ex. 2) Data set Input A B [17] C [27, 29] D [39, 37] E [49, 48, 47] F [59, 57, 58] G [67, 68, 69] H [77, 78, 79, 76] n Expected output ”no numbers” ” 17” ” 27 and 29” ” 37 and 39” ” 47 and 48” ” 57 and 58” ” 67 and 68” ” 76 and 77” = Actual output ”no numbers” ” 17” ” 27 and 0” ” 37 and 39” ” 47 and 48” ” 57 and 58” ” 67 and 0” ” 76 and 77” Debugging reveals that variable ”mi 2” erroneously retains initialization (0). Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [ 32 ] Feb 26, 2008

Debugging (Ex. 2) public static void main ( String[] args ) { int mi 1 = 0, mi 2 = 0; if (args. length == 0) /* 1 if-else */ System. out. println("No numbers"); else { mi 1 = Integer. parse. Int(args[0]); if (args. length == 1) /* 2 if-else */ System. out. println("Smallest = " + mi 1); else { int obs = Integer. parse. Int(args[1]); mi 2 = obs; /* 3 */ if (obs < mi 1) if { mi 2 = mi 1; mi 1 = obs; } for (int i = 2; i < args. length; i++) { /* 4 for */ obs = Integer. parse. Int(args[i]); if (obs < mi 1) /* 5 if-else */ { mi 2 = mi 1; mi 1 = obs; } else if (obs < mi 2) /* 6 if */ mi 2 = obs; } System. out. println("The two smallest are: " + mi 1 + " and " + mi 2); } } } Claus Brabrand, ITU, Denmark WHITE-BOX TESTING Re-Test: [ 33 ] Feb 26, 2008

Control Structures (cont’d 2) n switch: n n Choice points? do Stm while ( Exp ); Exp 1 ? Exp 2 : Exp 3 ”&&”; ”lazy conjunction” (aka. , ”short-cut ”): n n default : Stm* break; ”? : ”; ”conditional expression”: n n switch ( Exp ) { Swb* } do-while: n n Swb: case Exp : Stm* break; Exp 1 && Exp 2 ”||”; ”lazy disjunction” (aka. , ”short-cut ”): n Exp 1 || Exp 2 Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [ 34 ] Feb 26, 2008

Control Structures (cont’d 3) n try-catch-finally (exceptions): n n return / break / continue: n n return Exp ; break ; continue ; e. g. ; f(x) ”recursive method invocation”: n n return ; ”method invocation”: n n try Stm 1 catch ( Exp ) Stm 2 finally Stm 3 e. g. ; f(x) ”virtual dispatching”: n e. g. ; Claus Brabrand, ITU, Denmark f(x) WHITE-BOX TESTING [ 35 ] Feb 26, 2008

Outline n Motivation n n What is Testing? n n Methodology: ”coverage testing” Automation n n Relation to other programming-related tasks? White-box Testing n n Why bother with testing? How to automate testing Exercises n Training for the exam Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [ 36 ] Feb 26, 2008

Test Automation n (Re-)Running tests is boooring (& error prone) n n Thus, automate them ”once-and-for-all” JUnit: n public class My. Test. Case extends Test. Case { @Test // Testing if 3*2=6: public void test. Multiplication() { assert. Equals("Multiplication", 6, 3*2); } /*. . . other tests. . . */ } n Can be run from Eclipse/JUnit: n Claus Brabrand, ITU, Denmark (if appropriately ”subclassing” Test. Case) WHITE-BOX TESTING [ 37 ] Feb 26, 2008

Outline n Motivation n n What is Testing? n n Methodology: ”coverage testing” Automation n n Relation to other programming-related tasks? White-box Testing n n Why bother with testing? How to automate testing Exercises n Training for the exam Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [ 38 ] Feb 26, 2008

Exercise: n Part I: (produce) n Part II: (consume) Claus Brabrand, ITU, Denmark Warm up exercise: Draw a control-flow diagram for the ”do-while” construction § Program merge (in Java): List<Integer> merge(List<Integer> list 1, List<Integer> list 2); § Test your merge method: a) Label choice points b) Build ”coverage table” & make data set (test suite) c) Build ”expectancy table” d) Run test suite (upon failure: fix and retest program) § Introduce subtle bug § Run test to document presence of bug § Submit “erroneous program” to class program pool § Pick “erroneous program” from class program pool § Test merge program (and debug to find bug) § Re-Test fixed merge program § Write report (and send it to the teaching assistant) WHITE-BOX TESTING [ 39 ] Feb 26, 2008

Specification (merge) n Interface (for ”merge”): n List<Integer> merge(List<Integer> list 1, List<Integer> list 2); n I/O assumptions: n Input: both lists are sorted: n n Output: list must be sorted: n n …and, numbers occur maximum once (in each list) …and, numbers occur maximum once Programming constraints: no recursion ! n no java. util. Iterator’s ! n Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [ 40 ] Feb 26, 2008

The Testing Report n Report (ca. 3 pages): n Must explain how you tested, (debugged), & re-tested the erroneous ”merge” (it must include, at least): n n n i) The erroneous program ii) a ”Control-Flow Graph” for the program [hand-drawn ok] (incl. labelled ”choice points”) iii) ”Coverage Tables” iv) ”Expectancy Tables” Submit the testing report to the T. A. (Anders) (deadline: Tuesday; March, 11 at 09: 00 CET) n Note: the report is only on Part II n n Claus Brabrand, ITU, Denmark (i. e. , the program you didn’t write) WHITE-BOX TESTING [ 41 ] Feb 26, 2008

Example 1 public static void main ( String[] args ) { int mi, ma; if (args. length == 0) System. out. println("No numbers"); else { mi = ma = Integer. parse. Int(args[0]); for (int i=1; i < args. length; i++) { int obs = Integer. parse. Int(args[i]); if (obs > ma) ma = obs; else if (mi < obs) mi = obs; } System. out. println(”min=" + mi + ", " + "max=" + ma); }} Claus Brabrand, ITU, Denmark WHITE-BOX TESTING [ 42 ] Feb 26, 2008