Testing 29Apr2009 Petr Panuka petr panuskahp com QA

Testing 29/Apr/2009 Petr Panuška petr. panuska@hp. com QA Manager & Offshore Team Manager, SOA Center HP Software R&D © 2008 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice

Agenda 1. Why is SW testing necessary 2. Testing principles 3. Testing design techniques

Why is SW testing necessary • SW does not always work as we wish • Defects in SW can cause harm − Not all defects in SW result in a failure • Testing reduces the probability of undiscovered defects remaining in the software • Testing gives us the confidence about the tested SW

Testing Principles

Testing Principles • ISTQB defines 7 Testing Principles 1. 2. 3. 4. 5. 6. 7. Testing shows presence of defects Exhaustive testing is impossible Early testing Defect clustering Pesticide paradox Testing is context dependent Absence-of-errors fallacy

Testing shows presence of defects • Tests show the presence not the absence of defects. • Testing can’t prove that SW is defect free • Testing reduces the probability of undiscovered defects remaining in the software • However, the fact that no defects were found, is not a proof of correctness.

Exhaustive testing is impossible • It is not cost-effective − Defects are not of an equal risk • We need to prioritize our tests − Use risk analysis and priorities to focus testing efforts − Use appropriate testing techniques to meet the risk, giving us confidence in what testing has been performed • Exhaustive testing is equivalent to Halting problem − There is no algorithm that can for an arbitrary program and its input validates if the program works (with the given input) fine or not

Early testing • In SW development, what can be tested − Requirement specification − Use-case Analysis document − Technical Analysis document − Design document − Functional Implementation − Performance − Usability • What can be also tested − Test Description − Documentation Price of potential defect fix increases

Example

Example – Early testing • Defect in H 310: JBoss 4. 3. 0 not supported on Win 2008 (although Hermes 310 PRD requires this combination to be supported) − 7 other duplicates reported − Involving 4 people from QA − 9 people from DEV

Defect clustering • Defects get clustered since different reasons − A component might be more complex than others − A component might be developed by a less experienced developer − A component might be developed by a less careful developer − A component might have a poorer specification − A component might need more refactoring (introducing more other defects) • Another explanation: http: //parlezuml. com/blog/? postid=242

Pesticide paradox • Old tests will eventually stop finding new defects − Defects create immunity to these tests • To find defects, new tests need to be introduced − Or old tests refactored • Conclusion: Regression tests do not find majority of new defects

Testing is context dependent • Different kinds of tests are run in different periods − Defect testing • To discover faults or defects in the software where its behavior is incorrect or not in conformance with its specification; • A successful test is a test that makes the system perform incorrectly and so exposes a defect in the system. − Validation testing • To demonstrate to the developer and the system customer that the software meets its requirements; • A successful test shows that the system operates as intended. − We start with defect testing and perform validation testing later in the SW development process.

Testing is context dependent II. • We can also test differently because of − The type of industry (safety-critical, business, nuclear) − Number of customers (impact the SW makes) • One customer patch • New version of SW for potentially many customers

Absence-of-errors fallacy

Absence-of-errors fallacy • We test a system to see if it meets the documented requirements • We find and fix defects to demonstrate that the system meets these specifications • Finding and fixing defects does not help if the system is unusable and does not fulfill the user’s needs and expectations

Testing Techniques

Testing design techniques • Black-box techniques − Equivalence partitioning − Boundary value analysis − Decision tables − State transitions • No details about the product implementation are known White-box techniques − Statement testing and coverage − Decision testing and coverage − Other structural-based coverage Tester knows how the tested requirement is implemented

Equivalence partitioning • Tax Calculator based on annual income • Input – integral number • Income Tax < $20000 No tax $20000 - $50000 15% > $50000 20% Partitions of equivalence Invalid part Valid part #1 Valid part #2 Valid part #3 … -2, -1 0, 1, … 19999 20000, … 50000 50001, …

Equivalence partitioning II • Split input into partitions − Valid partitions − Invalid partitions • Have one test-case for each partition − Even invalid ones • One more example: Enter number of a month Invalid partition 1 Valid partition Invalid partition 2 … -2, -1, 0 1, 2, … 12 13, 14, …

Boundary value analysis • Similar as Equivalence partitioning but − Test boundary values • N-BVA (BVA = 0 -BVA) − Test N boundary values − BV, BV+1, BV-1, … BV+N, BV-N

Decision tables

Decision tables - Example • Example – HP SOA Systinet Licensing Framework • HP SOA Systinet consists of − Core, Reporting, Lifecycle, etc. − Policy Manager (optional) − Contract Manager (optional) • HP SOA Systinet License can limit − Number of users − Number of days • Default license included in the installer − Policy & Contract Manager included − Limited to 60 days, unlimited to number of users

Decision tables Example – Testing license application Actions Conditions •

Decision tables Conditions • Example – Testing license application Default PM on CM on # days limited # users limited Try PM Actions Try CM Login after expiration Login more users than allowed

Decision tables Conditions • Example – Testing license application Default PM on CM on # days limited # users limited Try PM Actions Try CM Login after expiration Login more users than allowed Y Y Y Y Y N N Y Y Y Y N N Y Y Y N N N Y Y Y Y N Y N Y Y N N Y Y Y N N N Y Y N N N Y Y Y N N Y Y N N N Y Y N Y N N N Y Y Y N N N Y N N N Y Y N N N N N Y N N N

Decision tables Actions Conditions • Example – Testing license application Default PM on CM on # days limited # users limited Try PM Y Y Y Try CM X Login after expiration Login more users than allowed X Default license Y Y N X Y Y Y N Y Y Y Y N N Y Y N Y N Y Y N N Y Y Y N N N Y N Y Y Y Y N Y N Y Y N N Y Y Y N N N Y Y N N N Y Y Y N N Y Y N N N Y Y N Y N N Y N N N Y Y Y N N Y Y N N N Y N N N Y Y N N N Y N N N Y N N N

Decision tables Actions Conditions • Example – Testing license application Default PM on CM on # days limited # users limited Try PM Y Y Y Try CM X Login after expiration Login more users than allowed X Default license Y Y N X Y Y Y N Y Y Y Y N N Y Y N Y N Y Y N N Y Y Y N N N Y N Y Y Y Y N Y N Y Y N N Y Y Y N N N Y Y N N N Y Y Y N N Y Y N N N Y Y N Y N N Y N N N Y Y Y N N Y Y N N N Y N N N Y Y N N N Y N N N Y N N N

Decision tables Actions Conditions • Example – Testing license application Default PM on CM on # days limited # users limited Try PM Y Y Y Try CM X Login after expiration Login more users than allowed X X Default license Y Y N X Y Y Y N Y Y Y Y N N Y Y N Y N Y Y N N Y Y Y N N N Y N Y Y Y Y N Y N Y Y N N Y Y Y N N N Y Y N N N Y Y Y N N Y Y N N N Y Y N Y N N Y N N N Y Y Y N N Y Y N N N Y N N N Y Y X N N N Y N N N Y N N N X Visibility Edition

Decision tables Actions Conditions • Example – Testing license application Default PM on CM on # days limited # users limited Try PM Y Y Y Try CM X X Login after expiration Login more users than allowed X X Default license Y Y N X Y Y Y N Y Y Y Y N N Y Y N Y N Y Y N N Y Y Y N N N Y N Y Y Y Y N Y N Y Y N N Y Y Y N N N Y Y N N N Y Y Y N N Y Y N N X N Y Y Unlimited Standard Edition N Y N N Y N N N Y Y Y N N Y Y N N N Y N N N Y Y X Visibility Edition N N N Y N N N Y N N N

Decision tables Actions Conditions • Example – Testing license application Default PM on CM on # days limited # users limited Try PM Y Y Y N X Y Y Y N Y Y Y Y N N Y Y N Y N Y Y N N Y Y Y N N N Y N Y Y Y Y N Y N Y Y N N Y Y Y N N N Y Y N N N Y Y X N Y Y Y N N Y Y N N X N Y Y N Y N N Y N N N Y Y Y N N Y Y N N N Y N N N Y Y X N N N Y N N N Y N N N Try CM X X X Login after expiration Login more users than allowed X X Default license Limited Standard Edition Unlimited Standard Edition Visibility Edition

Decision tables II • Catches all possible combinations • Helps to analyze the situation and to decide − Which use-cases must be tested − Which use-cases does not make sense − Requires knowledge of the business environment • Helps to prioritize the use-cases

State transitions • Example – Testing ‘Contract Request Lifecycle’ Create Request Accept Request Reject Request Revoke Request Delete Request

State transitions II • Transition diagram shows valid transitions − Does not show invalid (that should be also tested) • Good for testing use-cases that are possible to be described by transitions between states • The scenarios may contain test-cases for each − State − Transition − Event that triggers state change (transition) − Action that may result from those transitions

White-box techniques

Statement testing and coverage

Decision testing and coverage

Other structure-based techniques • Condition Coverage − All single boolean conditions in a single statement must be evaluated to true and false − Full condition coverage does not imply full decision one! • Condition/Decision Coverage − Hybrid metric composed by the union of condition coverage and decision coverage.

Other structure-based techniques II • Multiple Condition Coverage − All possible combinations of all boolean conditions in a single statement must be evaluated to true and false • Path Coverage − Whether each of the possible paths in each function have been followed. A path is a unique sequence of branches from the function entry to the exit. • Function, Call, LCSAJ, Loop, Race, etc. coverages • FMI, see http: //www. bullseye. com/coverage. html

Example • RTCA published DO-178 B that requires minimal coverage for aeronautics SW systems based on their criticality: Effect of System Failure Example Required Code Coverage Catastrophic Crash Hazardous Passenger fatality 100% condition/decision coverage 100% statement coverage 100% decision coverage 100% statement coverage Major Passenger injury 100% statement coverage Minor Flight plan change No code coverage requirement Entertainment system failure No code coverage requirement No effect

Summary

Summary • Testing Principles − Testing shows presence of defects − Exhaustive testing is impossible − Early testing − Defect clustering − Pesticide paradox − Testing is context dependent − Absence-of-errors fallacy • Testing Techniques − Black-box techniques − White-box techniques

Q&A