Chapter 2 BASICS OF SOFTWARE TESTING Aditya P

Chapter 2 BASICS OF SOFTWARE TESTING © Aditya P. Mathur 2005 02/06/12

Software and Hardware Testing [Contd. ] Test Coverage: Practically impossible to completely test a large piece of software (OS) or any complex integrated circuit (32 or 64 bit microprocessor). This leads to notion of Acceptable Test Coverage. A A S-a-1 O O B B Correct Incorrect © Aditya P. Mathur 2005

Software and Hardware Testing [Contd. ] Correct NAND Gate NAND gate with fault input in A abbreviated s-a-1 Correct NAND Gate Faulty NAND Gate A B 0 0 1 0(1) 0 1 1 0(1) 1 0 1 1(1) 0 1 1 1 0 1(1) 1 0 © Aditya P. Mathur 2005 O A B O

Testing and verification Program verification aims at proving the correctness of programs by showing that it contains no errors. This is very different from testing that aims at uncovering errors in a program. Program verification and testing are best considered as complementary techniques. In practice, one can shed program verification, but not testing. © Aditya P. Mathur 2005

Testing and verification (contd. ) Testing is not a perfect technique in that a program might contain errors despite the success of a set of tests. Verification might appear to be perfect technique as it promises to verify that a program is free from errors. Disadvantages of Verification : The person who verified a program might have made mistake in the verification process. There might be an incorrect assumption on the input conditions; incorrect assumptions might be made regarding the components that interface with the program, and so on. Verified and published programs have been © Aditya P. Mathur 2005 shown to be incorrect.

Defect Management Defect management is an integral part of development and test process. It is a subprocess of the development process. © Aditya P. Mathur 2005 Defect Prevention Defect Discovery Recording and Reporting Classification Resolution Prediction

© Aditya P. Mathur 2005 Defect Management [contd. ] Defect prevention is achieved by using good coding techniques, unit test plan, code inspection. Defect discovery is achieved by identifying defects, found in static and dynamic testing. Defects found are classified and recorded in database [ High Severity, Low Severity ] Defect prediction can be achieved by SOURCE CODE ANALYSIS before entering the testing phase, as how many defects might be in the program or the application. Some of the defect Management tools are Bugzilla, an opensource and Fogbugz.

EXECUTION HISTORY • Execution History of a program, also known as Execution trace, is an organized collection of information about various elements of a program during a given execution. • An execution slice is an executable subsequence of execution history. • E. g. t 1: <x=2, y=3> © Aditya P. Mathur 2005 t 2: <x=1, y=0>

© Aditya P. Mathur 2005 Test generation Any form of test generation uses a source document. In the most informal of test methods, the source document resides in the mind of the tester who generates tests based on a knowledge of the requirements. In most commercial environments, the process is a bit more formal. The tests are generated using a mix of formal and informal methods directly from the requirements document serving as the source. In more advanced test processes, requirements serve as a source for the development of formal models. 9

Test generation strategies Model based: require that a subset of the requirements be modeled using a formal notation (usually graphical). Models: Finite State Machines, Timed automata, Petri nets, etc. Specification based: require that a subset of the requirements be modeled using a formal mathematical notation. Examples: B, Z, and Larch. Code based: generate tests directly from the code. © Aditya P. Mathur 2005 10

Test generation strategies (Summary) © Aditya P. Mathur 2005 11

Static Testing © Aditya P. Mathur 2005

Static Testing Static testing is carried out without executing the application under test. Static testing helps discover faults, ambiguities and errors in the requirements and other application relation documents. Static testing is complementary to dynamic testing. Static testing is done by team of individuals who did not write the code.

Elements of static testing Static testing accesses the following Requirement documents Design documents User manuals Static testing tools Application code and documentation Requirements Control flow, Data flow and other data List of error recommendations

Walkthroughs Walkthrough is an informal process to review any application related document. Requirement walkthrough Code walkthrough ( peer code review ) Review plan User needs Free from ambiguities and inconsistencies Functional and nonfunctional requirements review Detailed report generation by understanding desired application.

Inspections Inspection is a formal process usually associated with code Formal code inspection is done to improve code quality at lowest cost The team works with the help of an inspection plan which consists of • Statement of purpose • Work product to be inspected ( Code & Document inspection) • Team formation ( Roles & Tasks to perform ) • Rate at which inspection task is to be completed • Data collection forms ( Consist defects discovered, Code standard violation and Time spent in each task )

Inspections [Contd. ] Inspection team members roles • Moderator ( Takes charge of process and leads the review ) • Reader ( Reads the code by using code browser and large monitor ) • Recorder ( Records any error discovered or issue to be looked into ) • Author ( Actual developer who helps other to understand the code ) 1. Static code analysis tools – CFG, Data flow graph 2. Software complexity and static testing

Model based testing and model checking Model based testing refers to the acts of modeling and generation of test from a formal model of application behavior Model checking refers to a class of techniques that allow the validation of one or more properties from a given model of an application Source: Requirements, Experience, Program Model checker Model Property satisfied ? Yes No Update model Or Source

Program representation: Control flow graphs 19

What is CFG? • CFG captures the flow of control within a program. • CFG assists testers in the analysis of a program to understand its behaviour in terms of flow of control. • CFG can be constructed manually for small programs. • CFG is also known as flow graph or program graph.

Program representation: Basic blocks • A basic block in program P is a sequence of consecutive statements with a single entry and a single exit point. • A block has unique entry and exit points. • Control always enters a basic block at its entry point and exits from its exit point. There is no possibility of exit or a halt at any point inside the basic block except at its exit point. • Entry point is the first statement and exit point is the last statement in a block. • The entry and exit points of a basic block coincide when the block contains only one statement. 21

Basic blocks: Example: Computing x raised to y 22

Basic blocks: Example (contd. ) Basic blocks © Aditya P. Mathur 2005 23

Control Flow Graph (CFG) • A control flow graph (or flow graph) G is defined as a finite set N of nodes and a finite set E of edges. • An edge (i, j) in E connects two nodes ni and nj in N. We often write G= (N, E) to denote a flow graph G with nodes given by N and edges by E. • Start and End are two special nodes in N, known as “Distinguished Nodes” • Every node is reachable from Start and every node in the path terminates at End. • Node start has no incoming edge and end has no outgoing edge. 24

Control Flow Graph (CFG) • In a flow graph of a program, each basic block becomes a node and edges are used to indicate the flow of control between blocks. • A pictorial representation of flow graph is often used in the analysis of control behaviour of a program. • Each node is represented by symbol usually an oval shape or rectangular box. These boxes are labelled using their corresponding block nos. • The blocks are connected by lines representing edges. Arrows are used to indicate the direction of flow. • The blocks that ends in a decision has two edges. 25

CFG(contd) • Blocks and nodes are labeled such that block bi corresponds to node ni. An edge (i, j) connecting basic blocks bi and bj implies that control can go from block bi to block bj.

CFG Example N={Start, 1, 2, 3, 4, 5, 6, 7, 8, 9, End} E={(Start, 1), (1, 2), (1, 3), (2, 4), (3, 4), (4, 5), (5, 6), (6, 5), (5, 7), (7, 8), (7, 9), (9, End)} 27

CFG Example Same CFG with statements removed. N={Start, 1, 2, 3, 4, 5, 6, 7, 8, 9, End} E={(Start, 1), (1, 2), (1, 3), (2, 4), (3, 4), (4, 5), (5, 6), (6, 5), (5, 7), (7, 8), (7, 9), (9, End)} 28

Paths Consider a flow graph G= (N, E). A sequence of k edges, k>0, (e_1, e_2, … e_k) , denotes a path of length k through the flow graph if the following sequence condition holds. Given that np, nq, nr, and ns are nodes belonging to N, and 0< i<k, if ei = (np, nq) and ei+1 = (nr, ns) then nq = nr. 29

Path(contd) • For nodes n, m of node set N, m is said to be descendant of n if there is path from n to m. ‘n’ is said to be ancestor of m. • The set of all successor and predecessor nodes of n will be denoted by succ(n) and pred(n) respectively. • ‘Start’ node has no ancestor and ‘End’ has no descendant. • Node can be its own ancestor or descendant.

Complete Path • A path through the flowgraph is considered to be complete if the first node along the path is start node and terminating node is end node.

Paths: feasible paths A path p through a flow graph for program P is considered feasible if there exists at least one test case which when input to P causes p to be traversed. p 1= ( Start, 1, 3, 4, 5, 6, 5, 7, 8, 9, End) p 2= (Start, 1, 1, 2, 4, 5, 7, 9, End) 32

Paths: sample paths through the exponentiation flow graph Two feasible and complete paths: p 1= ( Start, 1, 2, 4, 5, 6, 5, 7, 9, End) p 2= (Start, 1, 3, 4, 5, 6, 5, 7, 9, End) Specified unambiguously using edges: p 1= ( (Start, 1), (1, 2), (2, 4), (4, 5), (5, 6), (6, 5), (5, 7), (7, 9), (9, End)) Bold edges: complete path. Dashed edges: subpath. 33

Number of paths There can be many distinct paths through a program. A program with no condition contains exactly one path that begins at node Start and terminates at node End. Each additional condition in the program can increase the number of distinct paths by at least one. 34

Number of paths • Presence of loops also can increase the no of paths. Each traversal of the loop body adds a condition thereby increasing no. of paths.

Types of software testing 36

Types of testing One possible classification is based on the following five classifiers for dynamic testing: C 1: Source of test generation. C 2: Lifecycle phase in which testing takes place C 3: Goal of a specific testing activity C 4: Characteristics of the artifact under test C 5: Test Process 37

Classifier C 1: Source of Test Generation q Black Box Testing : • Tests generated from informally to formally specified requirements and without the aid of the code that is under test is known as Black box testing. • When requirements are informally specified, one could use adhoc techniques such as equivalence partitioning and boundary value analysis. • It is functional testing.

Classifier C 1: Source of Test Generation q Model Based or specification based testing – • Occurs when requirements are formally specified. • Example statecharts and flowcharts are generated using formal requirements. • This is also a form of black box testing

Classifier C 1: Source of Test Generation q White Box testing : • White box testing refers to the testing activity wherein code is used in the generation of the assessment of test case. • A test case consists of inputs and expected outputs, one must use requirements to generate testcases, the code is used as additional artifact. • There are techniques for generating testcases exclusively from code and expected output from requirement. • This from of testing is structural testing.

Classifier C 1: Source of Test Generation q White Box testing Code could be used directly or indirectly to generate the testcases. o In the direct case tool or human examines code and focuses on a given path to be covered. A test is generated to cover this path. o In the indirect case, tests generated using black box technique are assessed against some code coverage criteria. o Control flow and data flow can be used to generate testcases.

Classifier C 1: Source of Test Generation q Interface Testing • Tests are often generated using component’s interface. • The interface is a part of component’s requirement and hence this is form of black box testing. • Examples – Pairwise testing and interface mutation. • Pairwise testing – The set of values for each input is obtained from the component’s requirement. It is black box technique. • Interface mutation – Interface itself such as function coded in C serves to extract information needed to perform Interface mutation. It is white box technique.

Classifier C 1: Source of Test Generation • Adhoc testing – A tester generates tests from requirements but without the use of any systematic method. • Random testing – tester uses systematic method to generate tests.

C 1: Source of test generation 44

C 2: Lifecycle phase in which testing takes place • Testing activtities takes place throughout the software lifecycle. • Testing is categorized based on the phase it occurs 1. Unit testing – - The testing of code written in early phase and before it is integrated with system components. - Programmer focuses on unit or small component that has been developed. - The goal is to ensure that unit function correctly in isolation.

C 2: Lifecycle phase in which testing takes place 2) Integration testing – - When the units are integrated and a large component or subsystem forms, integration testing is carried out. - Goal of integration testing is to ensure that collection of components function as desired. - Most of Integration errors are discovered at this stage.

C 2: Lifecycle phase in which testing takes place 3) Sytem testing – - When the entire system is built system testing takes place. - Goal of system testing is to ensure that all desirwed functionality is in the system and works as per its requirements.

C 2: Lifecycle phase in which testing takes place 4) Beta Testing – - Often a Carefully selected set of customers is asked to test a system before commercialization. - In the case of contract software customer performs acceptability testing prior to making final decision as to whether to purchase the application for deployment.

C 2: Lifecycle phase in which testing takes place 5) Regression Test – - Goal of regression testing is to ensure that modified system functions as per the specification. - Regression testing may be performed using subset of the entire set of testcases used for system testing. - Testcases selected for regression testing include those designed to test the modified code and any other code that might be affected by modification.

C 2: Lifecycle phase in which testing takes place 50

C 3: Goal of specific testing activity • Goal oriented testing looks for specific types of failures. • Example – Vulnerability testing detects if there is any way by which the system under test can be penetrated by unauthorized users. 1) Robustness Testing - Refers to the task of testing an application for robustness against unintended inputs. - It differs from functional testing in that the tests for robustness are derived from outside of the valid input space whereas functional tests are derived from valid input space.

C 3: Goal of specific testing activity 2) Stress Testing – - One checks for the behavior of an application under stress. - examples 1) Handling of overflow data storage. 2) Web application can be tested by stressing them with large no. and variety of requests. - Conformance to functional as well as performance requirements under stress.

C 3: Goal of specific testing activity 3) Performance testing - Refers to that phase of testing where an application is tested specifically with performance requirements in view. - ex. Compiler performance in terms of lines of code compiled per sec. - Performance requirements are stated with respect to hardware and software cofiguration.

C 3: Goal of specific testing activity 4) Load Testing – - Refers to the phase of testing in which an application is loaded with respect to one or more operations. - The goal is to determine if the application continues to perform as required under various load condition. - Load testing can determine whether the application is handling exception in an adequate manner. - Ex. Handling out of memory exception. - Load testing is form of robustness testing i. e. testing for unspecified requirements. - It allows to find threshold limit of an application.

C 3: Goal of specific testing activity © Aditya P. Mathur 2005 55

C 4: Artifact under test • Testing techniques are named after the artifact being used. • Batch processing applications pose a special challenge in testing. • Ex: Payroll and student record processing application • Sometimes an application might not be batch processing application but no. of tests may need to be run in batches. Ex. Embedded application. • IBM’s Websphere is one of the several tool available for batch processing application.

C 4: Artifact under test 57

C 5: Test process models Testing in waterfall model Testing in V-model Spiral testing Agile testing Usually done toward the end of development cycle Explicitly specifies testing activities in each phase of development cycle Applied to software increments, proposed for evolutionary software development Used in agile development methods, such as e. Xtreme programming (XP) Test-driven development Requirements specified as tests (TDD)

Testing in waterfall model • The waterfall model is one of the earliest and least used software life cycle model. • While verification and validation of documents produced in each phase is an essential activity, static as well as dynamic testing occurs toward the end of process. • As the waterfall model adheres to sequential process, defects introduced in the early phase and discovered in the later phase could be costlier to correct. • There is very little iterative or incremental development when using waterfall model.

Testing in waterfall model Requirement specification Design Coding and unit testing System testing Acceptanc e testing Training and delivery Maintenance

Testing in V-model • Testing activities begin from the start and continue until end of lifecycle. • Test design begins soon after requirements are available.

Testing in V-model Maintenance Requiremen t analysis Design Coding Validate requirements Develop system acceptance test Validate design Develop integration test System testing Integratio n testing Develop unit tests unit testing Acceptanc e testing

Spiral testing • Spiral model and spiral testing are not same. • While testing is a key activity in the spiral model, spiral testing refers to the test strategy that can be applied to any incremental software development process especially where a prototype evolves into an application. • The sophistication of test activities increases with stages of an evolving prototype. • Early stage in Spiral testing focus on test planning. • Subsequent iterations refine requirements.

Spiral testing(Contd) • Once unit and integration test is performed, in final stage when requirements are well defined, testers focus on system and acceptance testing. • The cost of subsequent activities increases with subsequent iterations.

Spiral testing (cost of test activities increases with iterations) Intermediate test activities Initial test activities, early stages of development Cost Final test activities, application ready for acceptance test Time (prototype , evolves over time)

The saturation effect The saturation effect is an abstraction of a phenomenon observed during the testing of complex software system. The horizontal axis in the figure refers to the test effort that increases over time. The vertical axis refers to the true reliability and the confidence in the correct behavior of the application under test. The vertical axis can also be labeled as the cumulative count of failures that are observed over time, that is, as the test effort increases. The error correction process usually removes the cause of one or more failures.

Confidence and true reliability Confidence in the figure refers to the confidence of test manager in the true reliability of the application under test. An estimate of reliability , obtained using a suitable statistical method, can be used as a measure of confidence. 0 indicates lowest possible confidence and a 1 the highest.

Saturation region Suppose that application A is in the system test phase. The test team needs to generate tests, encapsulate them into appropriate scripts, set up the test environment, and run A against the test. The true reliability of A, with respect to the operational profile used in testing, increases as error are removed. Certainly the true reliability decreases in case that where fixing an error introduces additional errors-a case that we ignore.

False sense of confidence The discovery and fixing of previously undiscovered faults might increase our confidence in the reliability of A. It also increases the true reliability of A. In the saturation region when the expenditure of test does not reveal new faults, the reliability of A remains unchanged though our confidence is likely to increase due to a lack of observed failures. ∆1 in Figure is a measure of the deviation from the true reliability of A and a test manager’s confidence in the correctness of A.

Reducing delta ∆ Empirical studies reveal that every single test generation method has its limitations in the resulting test set is unlikely to detect all faults in an application. The more complex an application, the more unlikely it is that test generated using any given method will detect all faults. Now suppose that after having completed the test process using T 1 we check how much code in A has been exercised.

Reducing delta ∆ Enhancement of tests based on feedback received from adequacy assessment leads to tests that are guaranteed to move A to at least a few states that were never covered when testing using T 1.

Impact on test process A knowledge and appreciation of the saturation effect are likely to be of value to any test while designing and implementing a test process. Given that any method for the construction of functional tests is likely to lead to a test set that is inadequate with respect to code-based coverage criteria, it is important that test be assessed for their goodness.

Summary We have dealt with some of the most basic concepts in software testing. 73