1 Chapter 9 2 Software Testing Strategies Galin

1 Chapter 9. 2 Software Testing Strategies Galin, SQA from theory to implementation © Pearson Education Limited 2004

2 Chapter 9. 2 – Software Testing Strategies • • • Definitions and objectives Software testing strategies Software test classifications White box testing <<== • • • Data processing and calculation correctness tests Correctness tests and path coverage Correctness tests and line coverage Mc. Cabe’s cyclomatic complexity metrics Software qualification and reusability testing Advantages and disadvantages of white box testing Black box testing Equivalence classes for output correctness tests Other operation factor testing classes Revision factor testing classes Transition factor testing classes Advantages and disadvantages of black box testing Galin, SQA from theory to implementation © Pearson Education Limited 2004

3 Classification According to Testing Concept • Two main philosophies of testing: – One deals with internal structure of the software – The other deals with produced outputs given inputs. – IMHO, you definitely need both because they test the software from two entirely different perspectives! Galin, SQA from theory to implementation © Pearson Education Limited 2004

4 Definitions • Black Box (Functionality) testing: Identifies bugs only according to software malfunctioning as they are revealed in its erroneous outputs. – If outputs okay, BB Testing disregards internal paths taken, calculations and any processing. – This is somewhat of an oversimplification, but it is still essentially true. Galin, SQA from theory to implementation © Pearson Education Limited 2004

5 Definitions • White Box (Structural) testing: Examines internal paths program execution follows in order to identify bugs. – Often called ‘glass box testing’. – Investigates the correctness of the code structure. – Checks out the application in considerable detail – but with a cost. Galin, SQA from theory to implementation © Pearson Education Limited 2004

6 According to the IEEE: • Black Box Testing: – 1. Testing that ignores the internal mechanism of a system or component and focuses solely on the outputs generated in response to selected inputs and execution conditions. – 2. Testing conducted to evaluate the compliance of a system or component with specified functional requirements. • White Box Testing: – Testing that takes into account the internal mechanism of a system or component. – Checks out the paths and statements executed. Galin, SQA from theory to implementation © Pearson Education Limited 2004

7 • Clearly, black box testing is less expensive. • But it ‘clearly’ tests much less. • Some SQA requirements only advocate black box testing • Good, thorough testing almost always requires significant white box testing – and several of its derivatives – depending on the sensitivity of the software / component, as we shall discuss. Galin, SQA from theory to implementation © Pearson Education Limited 2004

8 Classification According to Requirements • Recall Mc. Call’s quality metrics. There are many tests and classification of tests that are hinged to requirements that Mc. Call’s metrics address. • Most test classifications dealing with requirements seem to loop back to one/more of Mc. Call’s quality metrics. • Some tests are ‘white box’ in nature, while others are ‘black box. ’ Some are complementary. • Let’s consider test classifications in more detail. Galin, SQA from theory to implementation © Pearson Education Limited 2004

9 White Box and Black Box Testing for classes of tests • Black Box testing: * We will look at these. – * Output correctness tests – * Documentation tests – * Availability (reaction time) tests – * Reliability tests – * Stress tests (load tests and durability tests) – * Software system security tests – Training usability tests – Operational usability tests – Flexibility tests – Testability tests – Portability tests – Software interoperability tests – Equipment interoperability tests Galin, SQA from theory to implementation © Pearson Education Limited 2004

10 White Box and Black Box Testing for classes of tests • White Box Testing: – * Data processing and calculations correctness tests – * Software qualification tests – * Reusability tests • Both white box and black box testing: – * Maintainability tests • Some of these overlap and are not necessarily mutuallyexclusive. Galin, SQA from theory to implementation © Pearson Education Limited 2004

11 White Box Testing - Overview • White box testing requires (can require) verification of every program statement / comment in your code – Can be very long and painstaking! • Calculation Correctness Test Cases are developed to test each computation and the path to the computation. – Used to determine the proper path sequence of steps that compute. • Software Qualification Checks out compliance and standards. • Maintainability checks out internal structure for maintenance • Reusability tests the reuse ability in a local software library. Galin, SQA from theory to implementation © Pearson Education Limited 2004

White Box Testing: In more Detail: Data Processing and Calculation Correctness. 12 • Enter ‘Coverage’ issues. – Major topics for many theses…. • Note: Cannot possibly cover every path. • Mathematically, there is an infinite number of paths. • So, we look at two categories: – Path coverage – Line coverage. Galin, SQA from theory to implementation © Pearson Education Limited 2004

13 White Box Testing "Path" vs “Line" coverage Path coverage • Path coverage is measured as percentage of all possible program paths included in planned testing. Line coverage • Line coverage is measured by the percentage of program code lines included in planned testing. • Tons of material in the literature on path, line, and branch coverage!! Galin, SQA from theory to implementation © Pearson Education Limited 2004

14 Correctness Tests and Path Coverage • Note: every path (if-then-else; do-while, etc. ) is a ‘different’ path with different values. • We’d like to tests all paths. • Total number of paths is countably infinite. • If one calculates the number of paths in a non-trivial routine, the number of test cases required is huge and very impractical. • Except for some special security applications, we will use alternative strategies: line coverage. Galin, SQA from theory to implementation © Pearson Education Limited 2004

15 Correctness Tests and Line Coverage • Desirable to test every line of code at least once during testing. • Define degree of coverage as the number of lines executed over the number of executable lines available. • Let’s consider a program flow chart to make our point. – a diamond is a decision; – a rectangle consists of imperatives and connections to conditional statements. – Edges represent sequences of sections • Let’s consider a program flow graph, with sections of code to replace one or more flow chart rectangles. Galin, SQA from theory to implementation © Pearson Education Limited 2004

Imperial Taxi Service Computations 16 ITS taxi fares for one-time passengers are calculated as follows: 1. Minimal fare: $2. This fare covers the distance traveled up to 1000 fic lights or traffic jams, etc. ) of up to 3 minutes. 2. For every additional 250 yards or part of it: 25 cents. 3. For every additional 2 minutes of stopping or waiting or part thereof: 4. One suitcase: 0 change; each additional suitcase: $1. 5. Night supplement: 25%, effective for journeys between 9 pm - 6 am. Regular clients are entitled to a 10% discount and are not charged the night Galin, SQA from theory to implementation © Pearson Education Limited 2004

17 Taxi Service - Flow chart - Path Coverage In planning, a flowchart and a flow graph were developed. Note: five decisions enter into planning for path testing… Twenty-four different paths may be executed to have full coverage. Means we need to prepare 24 different test cases. 1 Charge the minimal fare D > 1000 3 WT > 3 6 9 12 S >1 Yes 2 Distance 5 Waiting time 4 WT ≤ 3 7 8 No. of suitcases S≤ 1 11 Regular client? No 15 Yes 17 Print receipt. Galin, SQA from theory to implementation D ≤ 1000 14 Night journey? 10 13 No 16 © Pearson Education Limited 2004

Program flow graph 18 But, if we only insist on ‘line coverage, ’ we can get full line coverage with three tests: 1 1 2 3 5 6 8 9 11 12 17 1 2 4 5 7 8 10 11 13 14 15 17 1 2 4 5 7 8 10 11 13 14 16 17 2 3 So, we have three test cases compared with 24! Savings goes up exponentially with programs of greater complexity. 4 5 7 6 8 9 10 11 12 15 14 13 16 17 Galin, SQA from theory to implementation © Pearson Education Limited 2004

Program Flow Graph 19 Mc. Cabe’s cyclomatic complexity metrics also give an upper limit to the number of test cases needed for full line coverage. Mc. Cabe used graph theory to arrive at a minimum number of tests to ensure full line coverage based on a program flow graph. Any path must have at least one edge not in another path. 1 2 3 Galin, SQA from theory to implementation 4 R 2 7 5 6 8 10 R 3 9 Mc. Cabe developed his cyclomatic 12 complexity based on ‘regions’ in the graph. R 1 11 R 4 15 14 R 5 13 16 © Pearson Education Limited 2004

20 Mc. Cabe – Maximum Number of Independent Paths for Full Line Coverage. for full line coverage According to Mc. Cabe, the V(G) can be computed in any of three ways each of which is based on the program’s flow graph: V(G) = R // nbr of regions V(G) = E – N + 2 edges – nodes (21 -17 + 2 = 6) V(G) = P + 1 (nbr of decisions + 1) (5 + 1 = 6) R 6 Result: max nbr of independent paths Number of paths has been used to indicate overall program complexity 1 2 3 R 1 4 R 2 7 5 6 8 9 10 R 3 11 12 R 4 15 14 R 5 13 16 17 Galin, SQA from theory to implementation © Pearson Education Limited 2004

21 Software Qualification and Reusability Testing • Critically important for maintenance • Here we speak of coding and documentation according to local standards and procedures. – AFM 300 -4; AFDSDC 300 -8; – Data elements and codes; flowcharts; naming conventions for files, programs, manuals. • Reusability addresses – standards, – libraries of reusability modules, – tests, etc. Galin, SQA from theory to implementation © Pearson Education Limited 2004

22 Advantages and Disadvantages of White Box Testing Advantages: * Direct determination of software correctness as expressed in the processing paths statement by statement, including algorithms. Were they correctly defined and coded? ? Logic errors? ? Must carefully design test cases!! * Allows performance of line coverage follow up. We have software that identifies lines not yet executed are available. percent execute, phantom paths… * Ascertains quality of coding work and its adherence to coding standards. Galin, SQA from theory to implementation © Pearson Education Limited 2004

23 Advantages and Disadvantages of White Box Testing Disadvantages : * The vast resources utilized, much more those required for black box testing of the same software package. * The inability to test software performance in terms of availability (response time), reliability, load durability, etc. Book cites that white box testing should be limited to modules of high risk and high cost of failure – whatever that means! After programs have been maintained over the years, these general testing strategies are too simplistic. Galin, SQA from theory to implementation © Pearson Education Limited 2004