Software Design F 28 SD 2 LifeCycle Perspective

Software Design (F 28 SD 2): Life-Cycle Perspective - Part 2 Andrew Ireland School of Mathematical & Computer Sciences Heriot-Watt University Edinburgh Software Design F 28 SD 2 © Andrew Ireland

Outline • A strategy for dynamic analysis - testing • Designing test cases – specification and construction of test case data Software Design F 28 SD 2 © Andrew Ireland

Testing (Dynamic Analysis) • Testing involves system operation, i. e. code execution. • Testing that does not exploit the internal structure of the code is known as specification-based, functional, blackbox, behavioural, requirement-based, … • Testing that does exploit the internal structure of the code is known as structure-based, structural, white-box, glass-box, coverage based, … • Testing will typically involve the construction of additional code to facilitate the testing process. Software Design F 28 SD 2 © Andrew Ireland

A Classification of Test Case Design Techniques Specification-based Structure-based Experience-based Equivalence analysis Statement coverage Error guessing Boundary value analysis Decision coverage … Condition/Decision coverage … … The aim of design techniques is to provide a systematic basis for developing effective test cases, i. e. test cases that will provide a high yield in terms of defect detection. Software Design F 28 SD 2 © Andrew Ireland

A Strategy for Dynamic Analysis 1. Use specification and experience-based techniques to design initial test cases. 2. Use coverage metrics to determine the effectiveness of the initial test cases. 3. Use structure-based techniques to increase coverage if judged necessary. Note: Both positive (desirable behaviours) and negative (undesirable behaviours) test cases should be used. Software Design F 28 SD 2 © Andrew Ireland

Error Guessing • • Error guessing is driven by experience and will typically cover both functional and structural perspectives. To illustrate, given a software component to test that exploits dynamic storage allocation, then an experienced tester will typically focus upon the correct use of resource allocation and deallocation, e. g. seek out memory leakage defects. • By its very nature, error guessing is hard to teach. However, check lists designed around language features and related defects can play a useful role in the transfer of experience. Software Design F 28 SD 2 © Andrew Ireland

Structural/Functional Balance • While internal knowledge can greatly simplify the task of dynamic testing - over reliance on the actual code should be avoided, i. e. risk of designing test cases that demonstrate that the code does what the code does! • The primary source of test cases should be the functional specification. The process of designing tests based upon functional specification will typically expose many defects before coding gets underway. Software Design F 28 SD 2 © Andrew Ireland

Equivalence Partitioning • A technique that partitions the space of possible program inputs (and potentially outputs) into a finite set of equivalence classes – a set of data values for which the program will perform the same function • For example, consider a program that accepts numbers, and doubles the odd numbers – this suggests partitioning the inputs into two equivalences classes, i. e. odd and even Software Design F 28 SD 2 © Andrew Ireland

Equivalence Classes (EC) Even EC Odd EC 1 3 5 0 … 7 9 … 2 4 6 8 14 Equivalence partitioning can significantly prune the space of possible test cases - assuming the equivalence partitioning is performed correctly! Software Design F 28 SD 2 © Andrew Ireland

Valid and Invalid ECs • Parts of the input space which a program should accept give rise to Valid Equivalence Classes • Parts of the input space which a program should accept give rise to Invalid Equivalence Classes Not positive Positive 1 … 2 3 Software Design F 28 SD 2 0 4 5 -1 -2 … -3 -4 © Andrew Ireland

Some Guidance for Partitioning • Ranges: if a requirement specifies a range then three ECs are required, one valid and two invalid. Note that there always implementation dependent boundaries, e. g. 16 -bit integer gives rise to a partition bounded by 32767 and -32768. • Size: if a requirement specifies a size then three ECs are required, one • valid and two invalid, e. g. a four character password gives rise to a valid EC which contains all four character alpha numeric sequences, and two invalid ECs containing less-than and greater-than four character alpha numeric sequences Sets: if a requirement specifies a set then two equivalence classes are required, one valid (member) and one invalid (non-member). Software Design F 28 SD 2 © Andrew Ireland

Boundary Value Analysis Software Design F 28 SD 2 © Andrew Ireland

Test Case Effectiveness • • • When have we tested enough? Coverage metrics provide one of the most widely used approaches to judging the effectiveness of testing. A coverage metric (CM) represents the ratio of metric items executed at least once (EM) to the total number of metric items (TM): CM = EM/TM For example, if the code under test contains 1000 statements (TM), are your test cases execute 900 of the statements (EM), then you have achieved 90% statement coverage, i. e. CM = 0. 9 Coverage metrics are provide a criteria for specifying and constructing structural test cases. . . Software Design F 28 SD 2 © Andrew Ireland

Statement Coverage Definition: Every statement is executed at least once. Advantage: understandable & relatively simple to achieve 100% coverage. Disadvantage: Even with 100%, statement coverage is weak, e. g. 1. int* ptr = NULL; 2. if (flag) ptr = &result; 3. *ptr = 666; 100% statement coverage only requires that the execution path including flag set to true is achieved. statements flag lines 1, 2, 3 true Note that if flag is set to false, then the code crashes! That is, statement coverage misses the bug! Software Design F 28 SD 2 © Andrew Ireland

Decision Coverage • • • Definition: Every statement and every decision (if-statement, whilestatement, etc) is executed at least once. Advantage: easy to understand relatively simple to achieve 100% coverage and overcomes the deficiency of statement coverage. Disadvantage: 100% coverage does not mean that the deeper logical structure of the decision point will be adequately explored, e. g. if (condition 1 || condition 2) statement 1; else statement 2; case (condition 1 || condition 2) branch 1 true 2 false Note: no guarantee that the decision point will be tested when condition 2 is true … Software Design F 28 SD 2 © Andrew Ireland

Condition/Decision Coverage • • Definition: Every statement and every branch is executed at least once, with each condition within each branch taking on all possible values at least once. Advantage: Addresses the deficiency of decision coverage to some extent, i. e. explores the deeper structure of the decision point to some extent. • Disadvantage: Can still leave untested combinations of conditions, e. g. if (condition 1 && (condition 2 || condition 3)). . . case condition 1 condition 2 condition 3 branch 1 true 2 false Software Design F 28 SD 2 © Andrew Ireland

Multiple Condition Coverage Software Design F 28 SD 2 © Andrew Ireland

Multiple Condition Coverage if (condition 1 && (condition 1 || condition 2)) … case condition 1 condition 2 condition 3 branch 1 true 2 true false true 3 true false true 4 true false 5 false true false 6 false true false 7 false true false 8 false Software Design F 28 SD 2 © Andrew Ireland

Modified Condition/Decision Coverage (MC/DC) • Definition: Every condition within a decision is executed • to shown that it can independently effect the outcome of the decision. Advantage: A compromise requiring fewer cases than multiple condition coverage, i. e. minimum of N+1 and a maximum of 2 N cases where N denotes the number of • Boolean operands involved. Disadvantage: Time consuming to calculate. No real gains when N is small. Software Design F 28 SD 2 © Andrew Ireland

MC/DC Coverage if (condition 1 && (condition 1 || condition 2)) … case condition 1 condition 2 condition 3 branch 2 true false true 3 true false true 4 true false 6 false true false Cases 2 and 6 show independence of condition 1 • Value of condition 1 changed • Values of condition 2 and condition 3 fixed • Both true and false branches executed Software Design F 28 SD 2 © Andrew Ireland

MC/DC Coverage if (condition 1 && (condition 1 || condition 2)) … case condition 1 condition 2 condition 3 branch 2 true false true 3 true false true 4 true false 6 false true false Cases 2 and 4 show independence of condition 2 • Value of condition 2 changed • Values of condition 1 and condition 3 fixed • Both true and false branches executed Software Design F 28 SD 2 © Andrew Ireland

MC/DC Coverage if (condition 1 && (condition 1 || condition 2)) … case condition 1 condition 2 condition 3 branch 2 true false true 3 true false true 4 true false 6 false true false Cases 3 and 4 show independence of condition 3 • Value of condition 3 changed • Values of condition 1 and condition 2 fixed • Both true and false branches executed Software Design F 28 SD 2 © Andrew Ireland

MC/DC Coverage if (condition 1 && (condition 1 || condition 2)) … case condition 1 condition 2 condition 3 branch 2 true false true 3 true false true 4 true false 6 false true false • Cases 2 and 6 show independence of condition 1 • Cases 2 and 4 show independence of condition 2 • Cases 3 and 4 show independence of condition 3 Software Design F 28 SD 2 © Andrew Ireland

Test Case Specifications to Test Cases • Construct test cases for the following if-statement using Condition/Decision coverage: if !(closed) && ((n < max) || flag) … • Where max denotes the constant 100, closed, flag are Boolean variables and n is an integer variable. • Test case specification: case closed (n < max) flag branch TC 1 false true TC 2 true false • Test cases, e. g. • TC 1: closed == false, n == 99, flag true • TC 2: closed == true, n == 100, flag == false Software Design F 28 SD 2 © Andrew Ireland

Summary • Learning outcomes: – A strategy for testing – An understanding of the complementary nature of specification-based and structure-based testing methods – Equivalence partitioning & boundary value analysis – Coverage metrics for structure-based testing • Recommended reading: – “Software Testing: An ISTQB-ISEB Foundation Guide” (2 nd Edition). BCS, 2010. – “Software Testing in the Real World”, E. Kit, Addison-Wesley, 1995. – “Black Box Testing”, B. Beizer. Wiley & Sons, 1995. – “Applicability of modified condition/decision coverage to software testing'', J. J. Chilensky and S. P. Miller. Software Engineering J. 1994. Software Design F 28 SD 2 © Andrew Ireland

Summary • Recommended reading: – “Software Testing: An ISTQB-ISEB Foundation Guide” (2 nd Edition). BCS, 2010. – “Software Testing in the Real World”, E. Kit, Addison-Wesley, 1995. – “Black Box Testing”, B. Beizer. Wiley & Sons, 1995. – “Applicability of modified condition/decision coverage to software testing'', J. J. Chilensky and S. P. Miller. Software Engineering J. 1994. Software Design F 28 SD 2 © Andrew Ireland