ECE 453 CS 447 SE 465 Software Testing

ECE 453 – CS 447 – SE 465 Software Testing & Quality Assurance Instructor Kostas Kontogiannis 1

Overview èStructural Testing èIntroduction – General Concepts èFlow Graph Testing èDD-Paths èTest Coverage Metrics èBasis Path Testing èGuidelines and Observations èData Flow Testing èHybrid Methods èRetrospective on Structural Testing 2

Path Testing • Path Testing is focusing on test techniques that are based on the selection of test paths though a program graph. If the set of paths is properly chosen, then we can claim that we have achieved a measure of test thoroughness. • The fault assumption for path testing techniques is that something has gone wrong with the software that makes it take a different path than the one intended. • Structurally, a path is a sequence of statements in a program unit. Semantically, a path is an execution instance of the program unit. For software testing we are interested in entry-exit paths. 3

Path Testing Process • Input: – Source code and a path selection criterion • Process: – – – Generation of a CFG Selection of Paths Generation of Test Input Data Feasibility Test of a Path Evaluation of Program’s Output for the Selected Test Cases 4

Flow Graphs – Use in determining Paths for Testing - Revisited V(G) = 3 1 Basis set: 1, 2, 3, 4, 6, 7 1, 2, 3, 4, 5, 4, 6, 7 1, 2, 6, 7 x = z+5 z = 4*3 -y x>z x = z+5 z = 4*3 -y if(x > z) goto A; for( u=0; u < x; u++) { z = z+1; }; A: y = z + k 2 6 f R 1 t R 2 3 5 z = z+1 u++ t R 3 u=0 4 y = z+k f 7 u<x Example of a simple control flowgraph. 5

Program Graph to DD-Path Graph first 4 5 6 7 9 10 8 B 11 E 12 F 13 21 22 23 D C 14 G 15 20 16 17 18 19 O H I N J K L M last 6

Topological Paths • We can identify eight possible paths through the previous diagram 1. first – A – B – C – E – F – G – O – last 2. first – A – B – D – E – F – G – O – last 3. first – A – B – C – E – F – H – I – N – O – last 4. first – A – B – D – E – F – H – I – N – O – last 5. first – A – B – C – E – F – H – J – K – M – N – O – last 6. first – A – B – D – E – F – H – J – K – M – N – O – last 7. first – A – B – C – E – F – H – J – L – M – N – O – last 8. first – A – B – D – E – F – H – J – L – M – N – O – last 7

Feasible Paths • If we assume that the logic of the program dictates that “if node C is traversed, then we must traverse node H, and if node D is traversed, then we must traverse Node G” – These constraints will eliminate Paths 1, 4, 6 and 8 – We are left to consider four feasible paths: 2. first – A – B – D – E – F – G – O – last 3. first – A – B – C – E – F – H – I – N – O – last 5. first – A – B – C – E – F – H – J – K – M – N – O – last 7. first – A – B – C – E – F – H – J – L – M – N – O – last 8

• Let’s consider the Triangle program, its CFG, and its DD-Path graph. [Jorgensen: Software Testing A Craftsman’s Approach] 1. program triangle (input, output); 2. VAR a, b, c: integer; 3. Is. ATriangle: boolean; 4. BEGIN 5. writeln(“Enter integers: “); 6. readln(a, b, c); 7. Writeln(“Side A is: “, a, “Side B is: “, b, “Side C is: “, c); 8. IF(a <b+c) AND (b < a+c) AND (c<a+b) THEN 9. Is. ATriangle = True; 10. ELSE Is. ATriangle = False; 11. IF Is. ATriangle) 12. THEN 13. BEGIN 14. IF (a=b) XOR (a=c) XOR (b=c) AND NOT((a=b) AND (a=c) 15. THEN writeln(“Equilateral”); 16. IF (a=b) AND (b=c) 17. THEN writeln(“Equilateral”); 18. IF(a <> b) AND (a <> c) AND (a<>c) 19. THEN writeln(“Scalene”); 20. END 21. ELSE Writeln(“Not a Triangle”); 22. END 4 8 9 10 11 5 6 14 15 16 17 18 19 7 21 first 20 22 A A B E C D G F I H L P 1: A-B-D-E-G-I-J-K-Last P 2: A-C-D-E-G-I-J-K-Last P 3: A-B-D-L-Last P 4: A-B-D-E-F-G-I-J-K-Last P 5: A-B-D-E-F-G-H-I-J-K-Last P 6: A-B-D-E-F-G-H-I-K-Last 13 12 K J last 9

Node B is traversed D, E are traversed Node C is traversed D, L are traversed Node E is traversed F, H, J are traversed Node F is traversed H, J are traversed Node H is traversed F, J are traversed Node J is traversed F, I are traversed • 1. program triangle (input, output); 2. VAR a, b, c: integer; 3. Is. ATriangle: boolean; 4. BEGIN A 5. writeln(“Enter integers: “); A 6. readln(a, b, c); A 7. Writeln(“Side A is: “, a, “Side B is: “, b, “Side C is: “, c); A 8. IF(a < b+c) AND (b < a+c) AND (c < a+b) THEN B 9. Is. ATriangle = True; C 10. ELSE Is. ATriangle = False; D 11. IF Is. ATriangle) D 12. THEN E 13. BEGIN E 14. IF (a=b) XOR (a=c) XOR (b=c) AND NOT((a=b) AND (a=c) F 15. THEN writeln(“Equilateral”); G 16. IF (a=b) AND (b=c) H 17. THEN writeln(“Equilateral”); I 18. IF(a <> b) AND (a <> c) AND (a<>c) J 19. THEN writeln(“Scalene”); K 20. END L 21. ELSE Writeln(“Not a Triangle”); 22. END first A A B E C D G F I H L J last Let’s consider the Triangle program, its CFG, and its DD-Path graph. [Jorgensen: Software Testing A Craftsman’s Approach] Logically feasible paths: K P 1: A-C-D-L-Last P 2: A-B-D-E-F-G-I-K-Last P 3: A-B-D-E-G-H-I-K-Last P 4: A-B-D-E-G-I-J-K-Last 10

Structured Programming Constructs s 2 s 3 s 2 s 1 s 1 s 4 Sequence if-then-else s 2 s 3 s 2 s 4 s 5 s 3 case if-then s 1 s 2 loop-pre-test loop-post-test 11

Violations of Structured Programming Constructs s 1 s 1 s 0 s 2 s 3 Branching into loop Branching out of loop s 2 s 0 s 3 Branching into decision s 3 s 1 Branching out of decision 12

Essential Complexity • Essential complexity. This measures how much unstructured logic exists in a module (e. g. , a loop with an exiting GOTO statement) • The essential complexity can be computed on a program graph that has been condensed by collapsing structured programming constructs to a single node, until no more structured constructs can be found 13

Condensing A Program Graph first a a A A B b E C D d b H G F L I H L c c last K J last first a e first L EV(G) = 1 e d last 14

Test Coverage Metrics - Revisited Metric Description of Coverage C 0 Every Statement C 1 Every DD-Path C 1 P Every predicate to each outcome C 2 C 1 Coverage + loop coverage Cd C 1 Coverage + every dependent pair of DD-Paths CMCC Multiple condition coverage Cik Every program path that contains up to k repetitions of a loop (usually k=2) Cstat “Statistically significant” fraction of paths C∞ All possible execution paths 15

Some Fundamental Path Selection Criteria • Exercise every statement or instruction at least once (statement coverage) C 0 • Exercise every branch and case statement, in each direction, at least once (branch coverage) C 1 • Exercise each condition in a decision (condition coverage) • Exercise each condition in a decision with all possible outcomes C 1 P • Exercise every compound predicate outcome (MCC) CMCC • Exercise every path from entry to exit C∞ 16

Example 1 2 F 3 F 10 F T SCPath 1: 1 -2 -3(F)-10(F)-11 -13 SCPath 2: 1 -2 -3(T)-4(T)-5 -6(T)7(T)-8 -9 -3(F)-10(T)-12 -13 4 T 5 12 11 Statement Coverage C 0: T Branch or Decision Coverage C 1: F 6 T 13 F 7 T 8 9 BCPath 1: 1 -2 -3(F)-10(F)-11 -13 BCPath 2: 1 -2 -3(T)-4(T)-5 -6(T)7(T)-8 -9 -3(F)-10(T)-12 -13 BCPath 3: 1 -2 -3(T)-4(F)-10(F)-11 -13 BCPath 4: 1 -2 -3(T)-4(T)-5 -6(F)-9 -3(F)10(F)-11 -13 BCPath 5: 1 -2 -3(T)-4(T)-5 -6(T)-7(F)-93(F)-10(F)-11 -13 17

Condition Coverage • Requires that each condition in a compound decision takes on all possible outcomes at least once • Condition Coverage does not require that each decision takes on all possible outcomes • Example: if Not (A or B) then C Setting A = True, B = False, then A = False and B = True satisfies Condition Coverage, but statement C will never be executed 18

Multiple Condition Coverage • Requires that each possible combination of inputs be tested for each decision. • Example: “if (A and B)” requires 4 test cases: A = True, B = True A = True, B = False A = False, B = True A = False, B = False • For n conditions, 2 n test cases are needed, and this grows exponentially with n 19

Modified MCC • Requires that each condition be shown to independently affect the outcome of a decision • Generally, for n conditions, MC/DC requires only n + 1 test cases • Example: if (A or B) then C 3 test cases: A = True, B = False (A dominates) A = False, B = True (B dominates) A = False, B = False (false outcome) 20

Examples • To test if (A or B) A: T F B: F T F F • To test if (A and B) A: F T T B: T F T • To test if (A xor B) A: T T F B: T F T 21

Guidelines and Observations • Functional testing focuses on properties that are “too far” or disassociated from the source code being tested • Path testing focuses too much on the graph and not the logic of the code • Path testing very useful to measure the quality of the testing (coverage criteria) • Basis path testing is giving us a lower bound of how much testing is necessary • Path testing is providing a set of metrics that act as cross checks to functional testing – When a program path is traversed by several functional test cases we suspect redundancy – When we fail to attain DD-Path graph coverage, we know that there are gaps is the functional test cases • Coverage metrics can be used as – Setting minimum testing standards and as – Mechanism to selectively test portions of the code more rigorously than others 22

Guidelines and Observations • We can distinguish between – Specified program behavior (S), – Programmed behavior (feasible paths) (P), and – Topologically feasible paths (T) S P T 23

Guidelines and Observations • Region 1 contains specified behavior that is implemented by feasible paths – most desirable • Every feasible path is topologically possible so the parts 2, 6 of set P should be empty • Region 3 contains feasible paths that corresponds to unspecified behavior. Such extra functionality must be inspected • Regions 4 and 7 contain the infeasible paths. Region 4 is problematic as it corresponds to specified behavior that has almost been implemented (topologically possible, yet infeasible program paths) – coding errors • Region 7 refers to unspecified, infeasible, yet topologically possible paths – potential maintenance problem – could become Region 3 S P 2 6 5 1 4 3 T 7 24

Software Testing Standards • • • • • IEEE 1008 Software Unit Testing IEEE 1044 Classification for Software Anomalies ISO/IEC 12207 Information Technology--Software Life Cycle Processes ISO/IEC TR 15271 Guide for ISO/IEC 12207 (Software Life Cycle Processes) AECL CE-1001 -STD REV. 1 Standard for Software Engineering of Safety Critical Software BSI BS-7738 Specification for Information Systems Products Using SSADM (Structured Systems Analysis and Design Method) BSI BS-7925 -1 Software Testing - Vocabulary BSI BS-7925 -2 Standard for Software Component Testing DEF 00 -55 (Part 1)/1 The Procurement of Safety Critical Software in Defence Equipment-Requirements DIN VDE 0801 Principles for Computers in Safety-Related Systems ESA PSS-05 -01 Guide to the Software Engineering Standards, Issue 1 German Process-Model (V-Model) Software Life-Cycle Process Model IAEA TRS-372 Development and Implementation of Computerized Operator Support Systems in Nuclear Installations IEC 60601 -1 -4 Medical Electrical Equipment--Part 1: General Requirements for Safety-4. Collateral Standard: Programmable Electrical Medical Systems IEC 60880 Software for Computers in the Safety Systems of Nuclear Power Stations IEE 5 Software Inspection Handbook NCC STARTS Purchasers' Handbook--Procuring Software-Based Systems NRC NUREG/BR-0167 Software Quality Assurance Program and Guidelines RTCA DO-178 B/ED-12 B Software Considerations in Airborne Systems and Equipment Certification 25