Testing Principles Testing Objectives Testing is the process

Testing Principles

Testing Objectives • Testing is • the process of exercising a program • with the specific intent of finding errors • prior to delivery to the end user.

What Testing Shows? errors requirements conformance performance an indication of quality

Testing Principles 1. All the tests should be traceable to customer requirements 2. Tests should be planned 3. 80% of errors – uncovered, during testing 4. Testing should begin “in the small” & progress toward testing “in the large” 5. Testing should be conducted by – independent third party

A strategic Approach to Software Testing • Testing strategy provides a process, • that describes for the developer, quality analysis & the customer • The steps conducted as a part of testing • The testing strategy includes -

The testing strategy includes • • • Test planning Test case design Test execution Data collection Effectiveness evaluation

The Strategic Approach for S/W Testing Can Be • Before starting testing process, • Formal technical reviews must be conducted • Various components of the system are tested • Then interface is tested • i. e. entire computer based system is tested • Different testing techniques can be applied at different point of time

• The developer of the software conducts testing • For the large projects, independent test groups also assists the developers • Testing & debugging are different activities • That must be carried out in software testing • Debugging also lies within any testing strategy

Testing Strategy System engineering Analysis modeling Design modeling Code generation Unit test Integration test Validation test System test

Testing Strategy • We begin by ‘testing-in-the-small’ • And • move toward ‘testing-in-the-large’

Verification & Validation

verification • Set of activities • That ensure that • Software correctly implements a specific function

Strategic Issues

• Before the testing stars, specify the product requirements appropriately • Specify testing objective clearly • Identify categories of users for the software e & their role with reference to software system • Develop a test plan that emphasizes rapid cycle testing • Build robust software which can b e useful for testing • Use effective formal reviews before actual testing process begins • Conduct formal technical reviews to assess the test strategy & test cases • Throughout the testing process adopt the continuous development strategy

Testing Strategies for Conventional Software

1. Unit testing 2. Integration testing • Non-incremental integration big-bang approach • Incremental integration top down testing bottom up integration regression testing smoke testing 3. Validation testing acceptance testing alpha testing beta testing 4. System testing recovery testing security testing stress testing performance testing

• A unit is the smallest testable part of software. • It usually has one or a few inputs and usually a single output. • In procedural programming a unit may be an individual program, function, procedure, etc. • In object-oriented programming, the smallest unit is a method, which may belong to a base/ super class, abstract class or derived/ child class. • Unit testing frameworks, drivers, stubs, and mock/ fake objects are used to assist in unit testing.

1. Unit Testing • Individual components are tested independently • Driver and stub software used in unit testing • Driver – a program, that accepts the test data & prints the relevant results • Stub – a subprogram, that uses – module interfaces and performs minimal data manipulation If required

Unit Testing module to be tested results software engineer test cases

Unit Testing module to be tested interface local data structures boundary conditions independent paths error handling paths test cases

Unit Test Environment driver interface local data structures Module boundary conditions independent paths error handling paths stub test cases RESULTS

BENEFITS (Advantages) of Unit Testing • Unit testing increases confidence in changing/ maintaining code. • If good unit tests are written and if they are run every time any code is changed, we will be able to promptly catch any defects introduced due to the change. • Also, if codes are already made less interdependent to make unit testing possible, the unintended impact of changes to any code is less. • Codes are more reusable. In order to make unit testing possible, codes need to be modular. • This means that codes are easier to reuse.

• The cost of fixing a defect detected during unit testing is lesser in comparison to that of defects detected at higher levels. • Compare the cost (time, effort, destruction, humiliation) of a defect detected during acceptance testing or when the software is live. • Debugging is easy. When a test fails, only the latest changes need to be debugged. With testing at higher levels, changes made over the span of several days/ weeks/ months need to be scanned.

Integration Testing • is a level of software testing - where individual units are combined and tested as a group. • The purpose of this level of testing is - to expose - faults in the interaction between integrated units. • Test drivers and test stubs are used - to assist in Integration Testing.

• Definition by ISTQB • integration testing: • Testing performed to expose defects • in the interfaces and in the interactions • between integrated components or systems

• ANALOGY • During the process of manufacturing a ballpoint pen, the cap, the body, the tail and clip, the ink cartridge and the ballpoint are produced separately and unit tested separately. • When two or more units are ready, they are assembled and Integration Testing is performed. For example, whether the cap fits into the body or not.

• When is Integration Testing performed? • Integration Testing is performed after Unit Testing and before System Testing. • Who performs Integration Testing? • Either Developers themselves or independent Testers perform Integration Testing.

• APPROACHES • Big Bang is an approach to Integration Testing where all or most of the units are combined together and tested at one go. This approach is taken when the testing team receives the entire software in a bundle. So what is the difference between Big Bang Integration Testing and System Testing? Well, the former tests only the interactions between the units while the latter tests the entire system.

• Top Down is an approach to Integration Testing where top level units are tested first and lower level units are tested step by step after that. This approach is taken when top down development approach is followed. Test Stubs are needed to simulate lower level units which may not be available during the initial phases.

• Bottom Up is an approach to Integration Testing where bottom level units are tested first and upper level units step by step after that. This approach is taken when bottom up development approach is followed. Test Drivers are needed to simulate higher level units which may not be available during the initial phases.

• Sandwich/Hybrid is an approach to Integration Testing which is a combination of Top Down and Bottom Up approaches.

2. Integration Testing • Group of dependent components are tested together • Uncovers errors in – - Design and construction of software architecture. - Integrated functions and operation at system level - Interfaces and interactions between them - Resource integration and environmental integration

Integration Testing Strategies Options: • the “big bang” approach • an incremental construction strategy

Approaches Of Integration Testing • Non incremental integration - big-bang approach • Incremental integration - top down testing - Bottom up testing - Regression testing - Smoke testing

Non Incremental Integration (Big-bang Approach) • Steps – - All components are combined in advance The entire program is tested as a whole And - chaos – usually results - set of errors are tested as a whole once these errors are corrected, new ones appear this process continues infinitely -

• Advantages – - simple • Disadvantages – - Hard to debug - difficult to isolate errors while testing

Incremental Integration - top down integration testing Bottom up integration testing Regression testing Smoke testing

Top Down Integration A B F top module is tested with stubs G stubs are replaced one at a time, "depth first" C as new modules are integrated, some subset of tests is re-run D E

Bottom-Up Integration A B G drivers are replaced one at a time, "depth first" C D F E worker modules are grouped into builds and integrated cluster 42

Regression Testing • re-execution of some subset of tests that have already been conducted to ensure that changes have not propagated unintended side effects • Whenever software is corrected, some aspect of the software configuration is changed. • Regression testing helps to ensure that changes (due to testing or for other reasons) do not introduce unintended behavior or additional errors. • Regression testing may be conducted manually, by re-executing a subset of all test cases or using automated capture/playback tools.

Smoke Testing • A common approach for creating “daily builds” for product software • steps: – Software components that have been translated into code are integrated into a “build. ” • A build includes all data files, libraries, reusable modules, etc. – The build is integrated with other builds – and the entire product is smoke tested daily.

3. Validation Testing

Acceptance testing • Conducted to ensure that – • The s/w works correctly • In the user work environment • Can be conducted over a period of weeks or months

Types of Acceptance Testing • Alpha testing - complete s/w is tested by customer - under supervision of developer - at developer’s site - in controlled environment • Beta testing - complete s/w is tested by customer - without developer being present - at customer’s site - no control of developer

4. System Testing

• DEFINITION • System Testing is a level of the software testing where a complete and integrated software is tested. • The purpose of this test is to evaluate the system’s compliance with the specified requirements. • Definition by ISTQB system testing: The process of testing an integrated system to verify that it meets specified requirements.

• The system test is - a series of tests conducted to fully test the computer based system (As a single unit)

• When is it performed? - System Testing is performed - after Integration Testing and before Acceptance Testing. • Who performs it? - Normally, - independent Testers perform System Testing.

• ANALOGY • During the process of manufacturing a ballpoint pen, the cap, the body, the tail, the ink cartridge and the ballpoint are produced separately and unit tested separately. When two or more units are ready, they are assembled and Integration Testing is performed. When the complete pen is integrated, System Testing is performed

Types of System Tests • Recovery testing - System’s ability to recover from failures. - s/w is forced to fail & then it is verified - whether the system is properly recoverable or not - Reinitialization, checkpoint mechanisms, data recovery & restart - are verified • Security testing - Verifies the protection mechanism – unauthorized internal or external access - Or willful damage

• Stress testing - determines breakpoint of a system, to establish maximum service level - Software is tested at highest load - than normal working environment - i. e. system is executed for – resources in abnormal quantity, frequency or volume • Performance testing - runtime performance of the software is evaluated - resource utilization is measured - such as – CPU load, throughput, response time, memory usage - ex. Beta testing

Acceptance Testing

• DEFINITION • Acceptance Testing is a level of the software testing where a system is tested for acceptability. • The purpose of this test is to evaluate the system’s compliance with the business requirements and assess whether it is acceptable for delivery. • Definition by ISTQB • acceptance testing: Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system.

• ANALOGY • During the process of manufacturing a ballpoint pen, the cap, the body, the tail and clip, the ink cartridge and the ballpoint are produced separately and unit tested separately. • When two or more units are ready, they are assembled and Integration Testing is performed. • When the complete pen is integrated, System Testing is performed. • Once System Testing is complete, Acceptance Testing is performed so as to confirm that the ballpoint pen is ready to be made available to the end-users.

• METHOD • Usually, Black Box Testing method is used in Acceptance Testing. • Testing does not normally follow a strict procedure • and is not scripted but is rather ad-hoc.

• When is it performed? • Acceptance Testing is performed • after System Testing and before making the system available for actual use.

• Who performs it? • Internal Acceptance Testing (Also known as Alpha Testing) • is performed by members of the organization that developed the software • but who are not directly involved in the project (Development or Testing). • Usually, it is the members of Product Management, Sales and/or Customer Support.

• External Acceptance Testing • is performed by people who are not employees of the organization that developed the software. – Customer Acceptance Testing is performed by the customers – User Acceptance Testing (Also known as Beta Testing) – is performed by the end users of the software. – They can be the customers themselves or the customers’ customers.

Testing Strategies For Object-Oriented Software

• Basic unit of testing is - a class - That contains attributes & operations • These classes integrate - To form object oriented architecture • These are called collaborating classes

• Unit testing, in OO context • Integration testing, in OO context

Unit Testing • Classes and objects are tested - By considering data attributes & corresponding operations. encapsulation – is tested. derived classes (inherited) are tested. polymorphism is tested.

Integration Testing • Thread based testing – - Integrates all classes that respond to one input or event for the system. - Each thread is then tested individually • Use based testing – - Independent classes are tested first - and then, dependent classes are tested

Validation Testing • Conventional Black box testing

OO Testing Strategy • class testing is the equivalent of unit testing – operations within the class are tested – the state behavior of the class is examined • integration applied three different strategies – thread-based testing— – integrates the set of classes – required to respond to one input or event – use-based testing— – integrates the set of classes – required to respond to one use case – cluster testing— – integrates the set of classes – required to demonstrate one collaboration

OO Testing Methods • Conventional Test Case Design Methods White box testing – basis path testing and loop testing Black box testing - use cases • Fault Based Testing - valid faults are identified - Integration testing detects – three types of faults wrong operation used unexpected result incorrect invocation • Scenario Based Testing many interaction errors are uncovered by building the scenarios

Testing strategies for - webapps

Web. App Testing I • The Content Model for the Web. App - contents are evaluated Syntactically & Semantically - syntactic errors – like – spellings, grammar etc. are assessed. - semantic errors – correctness, consistency & ambiguity are assessed. • The Interface Model is reviewed to ensure that all use cases can be accommodated. • The design model for the Web. App is reviewed to uncover navigation errors.

• The User Interface - is tested to uncover errors - in presentation and/or navigation mechanics. • Each functional component is unit tested.

Webapp Testing II • Navigation throughout the architecture is tested. • The Web. App is implemented in a variety of different environmental configurations and is tested for compatibility with each configuration. • Security tests are conducted in an attempt to exploit vulnerabilities in the Web. App or within its environment. • Performance tests are conducted.

• The Web. App is tested - by a controlled and monitored population - of end-users. - The results of their interaction with the are evaluated for content and navigation errors, system - usability concerns, compatibility concerns, and Web. App reliability and performance.

High Order Testing • Validation testing - Focus is on software requirements • System testing - Focus is on system integration • Alpha/Beta testing - Focus is on customer usage • Recovery testing - forces the software to fail in a variety of ways and verifies that recovery is properly performed • Security testing - verifies that protection mechanisms built into a system will, in fact, protect it from improper penetration • Stress testing - executes a system in a manner that demands resources in abnormal quantity, frequency, or volume • Performance Testing - test the run-time performance of software within the context of an integrated system

Debugging: A Diagnostic Process

Art Of Debugging

• Debugging – - is a process of removal of a defect - It occurs as a consequence of successful testing

Debugging Process 1. Test cases are executed 2. Actual results are compared with expected results 3. Find – the lack of correspondence between actual and expected results 4. The suspected causes are identified 5. And additional tests or regression tests are performed; To make the system work as per the requirement

The Debugging Process

Debugging Approaches (Techniques) • Brute Force Method – - “Let Computer Find The Error” approach is used - Memory dumps & runtime traces are examined - Program with right statements is loaded - To obtain clues to errors • Backtracking Method – - The source code is examined by looking backwards - From symptom - To potential causes of errors • Cause Elimination method – - binary partitioning method is used - to reduce number of locations - where errors can exist

Testing Vs Debugging Sr. No. 1 2 3 Testing Debugging process in which the bug is A process in which the bug or identified error is corrected by the programmer In testing, it is identified – In debugging, the root cause where the bug occurs of error is identified Starts with the execution of the test cases Starts after the testing process

Debugging Effort time required to correct the error and conduct regression tests time required to diagnose the symptom and determine the cause

Symptoms & Causes symptom may disappear when another problem is fixed cause may be due to a combination of non-errors cause may be due to a system or compiler error symptom cause may be due to assumptions that everyone believes symptom may be intermittent

Consequences of Bugs infectious damage catastrophic extreme serious disturbing mild annoying Bug Type Bug Categories: function-related bugs, system-related bugs, data bugs, coding bugs, design bugs, documentation bugs, standards violations, etc.

Correcting The Error • • Is the cause of the bug reproduced in another part of the program? In many situations, a program defect is caused by an erroneous pattern of logic that may be reproduced elsewhere. • • • What "next bug" might be introduced by the fix I'm about to make? Before the correction is made, the source code (or, better, the design) should be evaluated to assess coupling of logic and data structures. • • What could we have done to prevent this bug in the first place? This question is the first step toward establishing a statistical software quality assurance approach. If you correct the process as well as the product, the bug will be removed from the current program and may be eliminated from all future programs. •

Final Thoughts • Think -- before you act to correct • Use tools to gain additional insight • If you’re at an impasse, get help from someone else • Once you correct the bug, use regression testing to uncover any side effects