The Purpose of Testing The purpose of software

The Purpose of Testing The purpose of software testing is to assess and evaluate the quality of work performed at each step of the software development process. Although it sometimes seems that way, the purpose of testing is NOT to use up all the remaining budget or schedule resources at the end of a development effort. The goal of testing is to ensure that the software performs as intended, and to improve software quality, reliability and maintainability.

Testing doesn’t ensure quality The purpose of software testing is to assess or evaluate the capabilities or attributes of a software program’s ability to adequately meet the applicable standards and customer needs. So, when I hear a tester say the purpose of testing is to ensure quality I immediately ask them, “how can testing ensure quality? ” Think about this rhyme for a moment: ◦ ◦ I don’t get to drive the train, I don’t get to ring the bell, But, if the train jumps the tracks See who catches hell!

I don’t get to discuss the needs of the customer, I don’t get to translate those needs into requirements, I don’t get to design or code the solution, and when I find a bug in the software I don’t even get to fix it. I can’t even demand that every bug I find gets fixed. In fact, I know that a number of reported bugs (including ones I think should get fixed) simply won’t get fixed. So, how in the hell can I ensure quality? What I can do as a professional tester is to design the appropriate tests to evaluate the attributes and capabilities of a software program’s ability to handle valid and invalid inputs under various conditions. This approach not only has the potential to demonstrate a program’s ability to meet customer needs and expectations, but also includes identification of flaws or deficiencies that could negatively impact the customer (or long term maintenance of the product).

The purpose of testing is not to find bugs The primary objectives of testing include: Qualifying a software program’s quality by measuring it’s attributes and capabilities against expectations and applicable standards And (probably the most important role of testing is simply to) provide information

The purpose of Testing What We Do? Productivity and Quality in Software. Goals for Testing. ◦ Bug Prevention ◦ Bug Discovery Phases in a Tester’s Mental Life. ◦ ◦ ◦ ◦ Why Testing? Phase 0 Thinking – Phase 1 Thinking – Phase 2 Thinking – Phase 3 Thinking – Phase 4 Thinking – Cumulative Goals Testing and Debugging The Software Works. The Software Doesn’t Work Test for Risk Reduction A state of Mind Test Design Testing isn’t Everything

The other major methods in decreasing order of effectiveness are as follows Inspection Methods Design style Static analysis methods Languages Design methodologies and development envirnoment The Pesticide Paradox and the Complexity Barrier First Law: The Pesticide Paradox – Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual. Second Law: The complexity Barrier – Software complexity grows to the limits of our ability to manage that complexity.

Some Dichotomies Testing Versus Debugging Function Versus Structure The Designer Versus the Tester Modularity Versus Efficiency Small Versus Large The Builder Versus the Buyer The The The builder buyer user tester operator

A Model for Testing The Project ◦ ◦ ◦ ◦ ◦ Application Staff Schedule Specification Acceptance test Personnel Standards Objectives Source History

The World The Environment The Program Nature and Psychology The Model World Environment Model Program Model Tests Bug Model A Model of Testing Out Com e Expe cted

Overview The Environment The Program Bugs Benign Bug Hypothesis – the bug related logic. Bug Locality Hypothesis – component bug. Eg: Syntax. Control Bug Dominance - the control flow. Code / Data Separation – the errors in coding. Lingua Salvator Est – language syntax and semantics. Corrections Abide – the mistaken belief. Silver Bullets – language, design method, representation. Sadism Suffices – methodology and techniques. Angelic Testers – testers are better at test design than programmers are at code design.

Tests Testing and Levels Unit, Unit Testing Component, Component Testing Integration, Integration Testing System, System Testing The role of Models

Playing Pool and Consulting ORACLEs Playing Pool Oracles ◦ Input / Output Oracle ◦ Sources of Oracles Kiddie Testing Regression Test Suites Purchased Suites and Oracles Existing Program

Is complete Testing Possible If the objective of testing were to prove that a program is free of bugs, then testing not only would be practically impossible, but also would be theoretically impossible. Three different approaches can be used to demonstrate that a program is correct ◦ Test based on Structure ◦ Test based on Function and ◦ Formal proofs of correctness. “We can never be sure that the specifications are correct. ” “No verification system can verify every correct program”. “We can never be certain that a verification system is correct”.

Consequence of bugs ◦ Importance of bugs Frequency Correction Cost (cost of discovery and cost of correction) Installation Cost Consequences Importance = (frequency * correction cost + Installation_cost + Consequential_cost). ◦ How bugs affect us ◦ Flexible severity rather than absolutes ◦ Nightmare list and when to stop testing

How Bugs Affect Us Some of the ten consequences are 1. Mild: misspelled output or misaligned print out 2. Moderate- the bug impacts about systems performance 3. Annoying-operators must use unnatural command sequences 4. Disturbing-handle legitimate transactions. Ex: Atm 5. Serious-it loses track of transactions 6. Very serious-instead of losing your paycheck. The system credits into another account or converts deposits into withdrawals. 7. Extreme-The problems are not limited to a few users and few transaction types. they are frequent usual cases. 8. Intolerable- unrecoverable corruption of the databases. 9. Catastrophic-system may fails. 10. Infectious-it affects the social and physical environment.

Flexible severity rather than absolutes Many programmers, testers and quality assurance workers have an absolutist attitude toward bugs. Quality can be measured as a combination of factors. ◦ Correction cost: cost of the correction ◦ Context and application dependency: same bugs with same symtoms. ◦ Creating culture dependency : whts important depends on the creators of the software and their cultural aspirations ◦ User Culture Dependency whats important depends on the user culture. ◦ Software development phase: severity depends on the development phase.

Nightmare list and when to stop testing Nightmare is nothing but software problems. How should you go about quantifying the nightmare? ◦ Following sub procedures. 1. List your Worst software nightmares. State them in terms of the symptoms they produce and how your user react to those symptoms. 2. Convert the consequences of each nightmare into a cost. Usually this is labor cost for correcting the nightmare. 3. Order the list from the costliest to the cheapest 4. Based on your experience, measured the data and to identify the nightmares.

5. For each nightmare, then you have developed a list of possible causative bugs. Order that list by decreasing probability. Importance of bug type=∑ Cij. P(bug type in nightmare) all nightmares 6. Rank the bug types in order of decreasing importance to you. 7. Design tests and design your quality assurance inspection process by using the methods. 8. If the test is passed, then some nightmares or parts of them go away. . if the test id failed, then a nightmare is possible. 9. Stop the testing when the probability of all nightmares has been shown.

Taxonomy of bugs A taxonomy is a structure for classifying the bugs in hierarchical categories. The major categories are requirements, features functionality, structure, data, implementation and coding, integration, system and software architecture and testing.

Requirements, features and functionality bugs Requirements and specifications : developed from them can be incomplete, ambiguos. They can be misunderstood or impossible to understand. Features can be added, modified, deleted. Feature bugs: can be wrong, missing, extra. A missing feature is the easiest to detect and correct. A wrong feature could have deep design implications. Extra features were once considered desirable. Feature interaction : providing clear, correct, implementable, and testable feature is not enough. Features usually come in groups of related features Specification & feature bug remedies: 1. short term support: Fully automatic test case generation is possible Long term support: This approach will reduce but not eliminate, specification of bugs.

Structural bugs control and sequence of bugs: It includes path left out, unreachable code, improper nesting of loops, loop termination criteria incorrect, missing process steps. path testing can be used to identify these bugs Logic bugs: bugs in logic. misunderstanding the sematicsof the order in which Boolean expression is evaluated. These bugs are part of logical processing not related to control flow, then they are categorized as processing bugs

Processing bugs: includes arithmetic bugs, algebraic, mathematical, algorithm and general processing. although these bugs are frequent 12%, they tend to be caught in good unit testing. Initialization bugs: forgetting to initialize the working space, registers, or data areas; A bug in the first value of a loop control parameter; accepting initial value without a validation check. The test methods are helpful for the test deign and for debugging initialization problems Data flow bugs: A data flow anomaly occurs when there is path along which we expect to do something unreasonable with data, such as using an uninitialized variable, or initializing twice without an intermediate use, modifying the data and not storing or using the result.

Data bugs include all bugs that arise from the specification of data objects, their formats, the number of such objects, and their initial values. To increase the proportion of the source statements devoted to data definition is a direct consequence of two factors. Dramatic reduction in the cost of main memory and disc storage The high cost of creating and testing software. Dynamic vs. static Dynamic data are transitory. A storage object may be used to hold dynamic data of different types, with different formats , attributes. Static data are fixed in form and content. They appear in the source code or database, either directly or indirectly. Ex: a number, string of characters.

2)information, parameter and control: static or dynamic data can serve in one of three roles or in a combination of roles. . 3)Data specification consist of three parts: contents, attributes and attributes Contents: actual bit pattern, character string Structure: the size and shape and numbers that describe the data object. Attributes: semantics associated with the contents of a data object. ex: integer, alphanumeric string.

Coding bugs Codings errors of all kinds can create any of the other kind of bugs Most common kind of coding bug and often considered are documentation bugs. These bugs are simple spelling errors or the result of poor writing. Giving good source syntax checking.

Interface integration and system bug External interfaces Internal interfaces Hardware architecture Software architecture Operating system Control and sequence bugs Resource management problems Integration bugs System bugs

External interfaces The external interfaces are the means used to communicate with the world This include devices are sensors, input terminals, printers and communication lines Through protocols we can communicate the world

Internal interfaces It is based on internal environment The internal interfaces are closely related to the implementation details. They are Protocol design bugs I/p & O/p format bugs Wrong subroutine call sequence Call parameter bugs Misunderstood entry

Hardware architecture Software bugs are related to hardware architecture Ex: address generation error I/o device operation or instruction error I/O device address error Misunderstood the device status code Device protocol error Incorrect interrupt handling

Operating system Program bugs related to the operating system are combination of hardware architecture and interface bugs, mostly caused by misunderstanding of what is the operating system does. The remedy for OS interface bugs is the same as for hardware bugs.

Software architecture Software architecture bugs are often the kind that are called interactive All test techniques are applicable to the discovery of s/w architecture bugs Some of the sample bugs are failure to block or unblock interrupts, register or memory were initialized or not initialized

Control and sequence bugs System level control and sequence bugs include ignored timing. Assuming that events occur in a specified sequence. Specifying wrong priority, program state, processing level, redundancy. Path testing is applicable for this type of bugs.

Resource management problems Wrong resources used Resources already in use Resources are not returned into the right pool. Complicated resource structures are often designed in a misguided attempt to save memory Memory is cheap and getting cheaper Remedies 1. To prevent the bugs test methods can be used in efficient manner 2. Resource management is to keep the resource structure is simple.

Integration bugs Integration bugs are bugs having to do with the integration of , and with the interfaces between working and tested components All methods are used to transfer data directly or indirectly between components and all methods by which component share data can host the integration bugs. Integration bugs do not constitute a big bug category(9%) they are expensive category because they are usually caught late.

System bugs It can coveing all types of bugs like Hardware bugs Software bugs operating system bugs All testing techniques can be useful at all levels. unit testing and the integration testing can be used in system testing