Test Case Design Methodologies Blackbox methods P Pavankumar

Test Case Design Methodologies (Black-box methods) P. Pavankumar, CSTE Moderator, cste_mumbai@yahoo. com http: //www. geekswithblogs. net/ppothuraju/ ppothu@yahoo. com

Agenda • Black-box testing • Test Case – principles • Test Case Design – Equivalence Partitioning – Boundary Value Analysis – Cause – Effect Graph – Error Guessing

Black-box testing • In a Black-box testing the tester is completely unconcerned about the internal behaviour and structure of the program (tester views the program as a black box). • The tester is only interested in finding circumstances in which the program does not behave according to its specifications. • Test data are derived solely from the specifications. • Black-box testing strategy is also called as data-driven or input/output-driven testing.

Black-box testing • To find all the errors in the program the criterion is exhaustive input testing. • Exhaustive input testing is the use of every possible input condition as a test case. • Exhaustive testing is next to impossible and therefore a test of any program must be necessarily incomplete (i. e. the testing cannot guarantee the absence of all errors). • The obvious strategy then is to try to reduce this incompleteness as much as possible.

Test Case – principles* • A good test case is one that has a high probability of detecting an as-yet undiscovered error. • A successful test case is one that detects an as-yet undiscovered error. • A necessary part of the test case is a definition of the expected output or result • Test cases must be written for invalid and unexpected, as well as valid and expected input conditions * Courtesy: The Art of Software Testing - Myers

Test Case Design • The most important consideration in software testing is the design or invention of effective test cases. • The reason for the importance of test-case design stems from the fact that “complete” testing is impossible. • Given constraints on time, cost etc. , the key issue of testing becomes What subset of all possible test cases has the highest probability of detecting the most errors? • The study of test-case-design methodologies supplies reasonable answers to the above question.

Equivalence Partitioning is a black-box design method where test cases are developed by partition the input domain of a program into a finite number of equivalence classes such that one can reasonably assume (but, of course, not be absolutely sure) that test of a representative value of each class is equivalent to a test of any other value. • That is, if one test case in an equivalence class detects an error, all other test cases in the equivalence class would be expected to find the same error. • Conversely, if the test case did not detect an error, we would expect that no other test case in the equivalence class would find an error (unless a subset of the equivalence class falls within any other equivalence class, since equivalence classes may overlap one another).

Equivalence Partitioning Input domain partitioned into four sub-domains. 2 1 3 4 Too many test inputs. Four test inputs, one selected from each sub-domain.

How to partition? • Inputs to a program provide clues to partitioning. – Example 1: • Suppose that program P takes an integer X as input. • For X< 0 the program is required to perform task T 1 and for X>=0 task T 2.

How to partition? (contd. ) • The input domain is prohibitively large because X can assume a large number of values. • However, we expect P to behave the same way for all X<0. • Similarly, we expect P to perform the same way for all values of X>=0. • We therefore partition the input domain of P into two subdomains.

Two sub-domains One test case: X=-3 Equivalence class X<0 X>=0 Equivalence class Another test case: X=15 All test inputs in the X<0 sub-domain are considered equivalent (there behaviour is same i. e. they are less than X). The assumption is that if one test input in this sub-domain reveals an error in the program, so will the others. This is true of the test inputs in the X>=0 sub-domain also.

Non-overlapping Partitions • In the previous example, the two equivalence classes are nonoverlapping. In other words the two sub-domains are disjoint. • When the sub-domains are disjoint, it is sufficient to pick one test input from each equivalence class to test the program.

Non-overlapping Partitions • An equivalence class is considered covered when at least one test has been selected from it. • In partition testing our goal is to cover all equivalence classes.

Overlapping Partitions • Example 2: – Suppose that program P takes three integers X, Y and Z. It is known that: • X < Y • Z > Y

Overlapping Partitions X<Y, Z<=Y X=2, Y=3, Z=1 X>=Y, Z<=Y X=15, Y=4, Z=1 X<Y, Z>Y X=3, Y=4, Z=7 X>=Y Z>Y Z<=Y X>=Y, Z>Y X=15, Y=4, Z=7

Overlapping Partitions – Test Selection • In this example, we could select 4 test cases as: – X=4, Y=7, Z=1 satisfies X<Y – X=4, Y=2, Z=1 satisfies X>=Y – X=1, Y=7, Z=9 satisfies Z>Y – X=1, Y=7, Z=2 satisfies Z<=Y • Thus, we have one test case from each equivalence class.

Overlapping Partitions – Test Selection • However, we may also select only 2 test inputs and satisfy all four equivalence classes: – X=4, Y=7, Z=1 satisfies X<Y and Z<=Y – X=4, Y=2, Z=3 satisfies X>=Y and Z>Y • Thus, we have reduced the number of test cases from 4 to 2 while covering each equivalence class.

Partitioning using non-numeric data • In the previous two examples the inputs were integers. One can derive equivalence classes for other types of data also. • Example 3: – Suppose that program P takes one character X and one string Y as inputs. P performs task T 1 for all lower case characters and T 2 for upper case characters. Also, it performs task T 3 for the null string and T 4 for all other strings.

Partitioning using non-numeric data X: lc, Y: not null X: UC, Y: not null X: lc X: UC X: lc, Y: null Y: not null X: UC, Y: null lc: Lower case character UC: Upper case character null: null string.

Non-numeric Data • Once again we have overlapping partitions. • We can select only 2 test inputs to cover all four equivalence classes. – These are: • X: lower case, Y: null string • X: upper case, Y: not a null string

Guidelines to Equivalence Partitioning • Input condition specifies a range: create one for the valid case and two for the invalid cases. – e. g. for a<=X<=b the classes are • a<=X<=b (valid case) • X<a and X>b (the invalid cases)

Guidelines (contd. . ) • Input condition specifies a value: create one for the valid value and two for incorrect values (below and above the valid value). e. g. (“select minimum of one and maximum of 6 employees from the list of employees”) -- identify one valid equivalence class and -- two invalid equivalence classes (no employees and more than 6 employees) This may not be possible for certain data types, e. g. for Boolean (yes / no).

Guidelines (contd. . ) • Input condition specifies a member of a set: create one for the valid value and one for the invalid (not in the set) value. e. g. type of the programming languages must be C++, Java, Perl, -- Identify a valid equivalence class for each and -- One invalid equivalence class (e. g. C#)

Equivalence Partitioning - Facts • Equivalence Partitioning is vastly superior to a random selection of test cases. • It uses the fewest test cases possible to cover the most input requirements. • Although some guidelines are given to assist in the identification of equivalence classes, doing so is mostly an intuitive process. • It overlooks certain types of high yield test cases that boundaryvalue analysis can determine.

Boundary-value Analysis Boundary Analysis is black-box technique that consists of developing test cases and data that focus on the input and output boundaries of a given function. • Experience shows that test cases that explore boundary conditions have a higher payoff than test cases that do not. • Boundary conditions are those situations directly on, above and beneath the edges of input equivalence classes and output equivalence classes.

Boundary-value Analysis Boundary-value analysis differs from equivalence portioning in two respects 1) Rather than selecting any element in an equivalence class as being representative, boundary-value analysis requires that one or more elements be selected such that each edge of the equivalence class is the subject to a test. 2) Rather than just focusing attention on the input conditions (input space), test cases are also derived by considering the result space (i. e. output equivalence classes).

Guidelines - Boundary-value Analysis • If an input condition specifies a range of values, write test cases for the ends of the range, and invalid input test cases for situation just beyond the ends. • For instance if the valid domain of an input value is 1. 0 and +1. 0 write test cases fro the situations -1. 0, -1. 001 and 1. 001

Guidelines - Boundary-value Analysis • If and input condition specifies a number of values, write test cases for the minimum and maximum number of values and one beneath and beyond these values. • For instance if an input file can contain 1 -255 records, write test cases for 0, 1, 255, and 256 records.

Guidelines - Boundary-value Analysis • Use guideline 1 for each output condition. • For instance, if a program computes the monthly tax deduction and if the minimum is rs. 0. 00 and the maximum is rs. 1000 write test cases that cause rs. 0. 00 and rs. 1000 to be deducted. • Also, see if it is possible to invent test cases that might cause a negative deduction or a deduction more rs. 1000

Guidelines - Boundary-value Analysis • Use guideline 2 for each output condition. • For instance, if an information retrieval system displays the most-relevant abstracts based on the input request, but never more than four abstracts, • write test cases such that the program displays zero, one and four abstracts • write a test case that might cause the program to erroneously display five abstracts.

Boundary-value Analysis - example For example, a program which edits credit limits within a given range (10000 – 15, 000) would have • three equivalence classes: Ø Less than 10, 000 (invalid) Ø Between 10, 000 and 15, 000 (valid) Ø Greater than 15, 000 (invalid) In same credit limit example, • boundary analysis would test: Ø Low boundary plus or minus one (9, 999 and 10, 001) Ø On the boundary (10, 000 and 15, 000) Ø Upper boundary plus or minus one (14, 999 and 15, 001)

Boundary-value Analysis - Facts • Boundary-value compliments Equivalence Partitioning • The technique on the surface sounds simple, but boundary conditions may be very subtle and hence identification of them requires a lot of thought. • One weakness of Boundary-value analysis and Equivalence Partitioning is that they do not explore combinations of input circumstances.

Cause-Effect Graphing • Cause Effect Graphing is a systematic method of generating test cases representing combinations of conditions. • The graph is actually a digital logic circuit (a combination logic network), but rather than using standard electronics notation, a somewhat simpler notation is used. • No knowledge of electronics is necessary other than an understanding of Boolean logic (i. e. understating the logic operators and, or, and not)

Cause-Effect Graphing • A CAUSE may be thought of as a distinct input condition, or an ‘‘equivalence class’’ of input conditions. • An EFFECT may be thought of as a distinct output condition, or a meaningful change in program state. • Causes and Effects are represented as Boolean variables and the logical relationships among them CAN (but need not) be represented as one or more Boolean graphs.

Cause-Effect Graphing • Basic cause-effect graph symbols a a b IDENTITY NOT a b b a OR C d b C AND

Cause-Effect Graphing • Process to derive test cases – The specification is divided into “workable” pieces. – The CAUSES and EFFECTS in the specification are identified. • Causes and Effects are identified by reading the specification word by word and underlining words or phases that describe causes and effects. • Each cause and effect is assigned a unique number.

Cause-Effect Graphing • Process to derive test cases – The semantic content of the specification is analyzed and transformed into a Boolean graph linking the causes and effects. This is the cause-effect graph. – The graph is annotated with constraints describing combinations of cause and/or effects that are impossible because of syntactic or environmental constraints. – By methodically tracing state conditions in the graph is converted into a limited-entry decision table. Each column in the table represents a test case. – The columns in the decision table are converted into test cases.

Cause-Effect Graphing - Example • The character in column 1 must be an “A” or a “B”. The character in column 2 must be digit. In this situation the file update is made. If the first character is incorrect, message X 12 is issued. If the second character is not a digit, message X 13 is issued. The causes are: 1 – character in column 1 is “A” 2 - character in column 1 is “B” 3 - character in column 2 is a digit And the effects are: 70 – update made 71 – message X 12 is issued 72 – message X 13 is issued

Cause-Effect Graphing • Cause-Effect graph of the example 70 1 11 2 3 71 72 • Notice the intermediate node 11 that was created. • The graph represents the specification by setting all possible states of the causes and seeing that the effects are set to the correct values. Do achieve this we create dummy nodes like 11.

Cause-Effect Graphing • Cause-Effect graph of the example 70 1 11 2 3 71 72 • Notice the intermediate node 11 that was created. • The graph represents the specification by setting all possible states of the causes and seeing that the effects are set to the correct values. Do achieve this we create dummy nodes like 11.

Cause-Effect Graphing • Cause-Effect graph - apply the constraints – In most programs, certain combinations of causes are impossible because of syntactic or environmental considerations (e. g. a character cannot be an “A” and a “B” simultaneously). To account such scenarios we use constraints • Exclusive Constraints (at most one of ‘a’ and ‘b’ can be 1) • Inclusive Constraint (at least one of ‘a’, ‘b’, and ‘c’ must always be 1) • One and Only One (one and only one of ‘a’ and ‘b’ must be 1) • Requires (for ‘a’ to be 1, ‘b’ must be 1) • Masks (constraint among effects – if effect ‘a’ is 1, effect ‘b’ is forced to 0)

Cause-Effect Graphing • Cause-Effect graph - apply the constraints – Constraints Symbols Exclusive a Inclusive E b a b I C

Cause-Effect Graphing • Cause-Effect graph after applying Exclusive constraints 70 1 E 11 2 3 71 72

Cause-Effect Graphing • Cause-Effect graph after applying Exclusive constraints 70 1 E 11 2 3 71 72

Cause-Effect Graphing - Example Designing a Decision Table from the Cause-Effect Graph 1 Character A CAUSES Character B Digit Update made EFFECTS Message X 12 Message X 13 2 3 4 5

Cause-Effect Graphing • Designing a Decision Table from the Cause-Effect Graph ≠A A 1 70 ≠B E 11 2 B 71 Di ≠ Di 3 72

Cause-Effect Graphing - Example Designing a Decision Table from the Cause-Effect Graph Possible Values 1 2 1 ≠ 1 ≠ Character A CAUSES Character B Digit Update made EFFECTS Message X 12 Message X 13 3 4 5

Cause-Effect Graphing - Example Designing a Decision Table from the Cause-Effect Graph Applying Relationships / Combinations 1 Character A or B 1 CAUSES Character B Digit 1 Update made X EFFECTS Message X 12 Message X 13 2 3 4 5

Cause-Effect Graphing - Example Designing a Decision Table from the Cause-Effect Graph Applying Relationships / Combinations 1 2 1 ≠ 1 1 Character A or B CAUSES Character B Digit Update made X EFFECTS Message X 12 X Message X 13 3 4 5

Cause-Effect Graphing - Example Designing a Decision Table from the Cause-Effect Graph Applying Relationships / Combinations 1 2 3 1 ≠ 1 1 1 ≠ Character A or B CAUSES Character B Digit Update made X EFFECTS Message X 12 X Message X 13 X 4 5

Cause-Effect Graphing - Example Designing a Decision Table from the Cause-Effect Graph Each Column is one Test Case. 1 2 3 1 ≠ 1 Digit 1 1 ≠ Update made X Character A or B CAUSES EFFECTS Character B Message X 12 Message X 13 X X

Cause-Effect Graphing - Example Designing a Decision Table from the Cause-Effect Graph Each Column is one Test Case. 1 2 3 1 ≠ 1 Digit 1 1 ≠ Update made X Character A or B CAUSES EFFECTS Character B Message X 12 Message X 13 X X

Cause-Effect Graphing - Example Designing a Decision Table from the Cause-Effect Graph Each Column is one Test Case. 1 2 3 1 ≠ 1 Digit 1 1 ≠ Update made X Character A or B CAUSES EFFECTS Character B Message X 12 Message X 13 X X

Cause-Effect Graphing - Facts • Since cause-effect graphing requires the translation of a specification into a Boolean logic network, it gives one a different perspective on, and additional insight into, the specification. It’s a good way to uncover ambiguities and incompleteness in specifications. • Cause-Effect graph does not adequately explore boundary conditions. • The raise in the Causes and Effects increases the complexity of drawing the graph • The most difficult aspect of the technique is the conversion of the graph into the decision table. Improper conversion can lead to inaccurate test cases.

Error Guessing is bases on theory that test cases can be developed based upon the intuition and experience of the Test Engineer. For example if an input is the date, a test engineer may try February 29, 2000 or 9/9/0099

A Reasonable Strategy • The test-case design methodologies discussed can be combined into an overall strategy. The reason for combining them should be obvious by now; each contributes a particular set of useful test cases, but none of them by itself contributes a thorough set of test cases. • Again, the use of this strategy will not guarantee that all errors will be found, but it help in eliminating maximum errors.

Thank you P. Pavankumar, CSTE Moderator, cste_mumbai@yahoo. com http: //www. geekswithblogs. net/ppothuraju/ ppothu@yahoo. com