Lecture 5 Test Design Techniques Test Design Techniques

Lecture 5 Test Design Techniques

Test Design Techniques 5. 1 The Test Development Process 5. 2 Categories of Test Design Techniques 5. 3 Specification based or Black Box Techniques 5. 4 Structure based or White Box Techniques

5. 1 The Test Development Process • Developing test material can be split into two distinct stages: – Defining “what” needs to be tested – Defining “how” the system should be tested • This process can vary from organisation to organisation, can be very formal or very informal with little documentation • The more formal, the more repeatable the tests, but it does depend on the context of the testing being carried out • The Test Development Process consists of the following steps: 1. Defining test conditions 2. Specifying test cases 3. Specifying test procedures 4. Developing a Test execution schedule

Test Conditions, Cases, Procedures and Schedule What How Sourced Documentation Test Procedure Specification Test Condition Priority Test Cases Manual Test Script or Automated Test Script When Test Execution Schedule Test Procedure Specifications

5. 2 Categories of Test Design Techniques Definition of Black Box Testing What is black box testing? Definition of White Box Testing What is white box testing? Black, White and Experience based (comparison)

Black and White Box Testing Definition of Black Box Testing • Black-box testing: Testing, either functional or non-functional, without reference to the internal structure of the component or system.

Black and White Box Testing What is Black Box Testing? – testing without knowing the internal workings of the code – WHAT a system does, rather than HOW it does it – typically used at System Test phase, although can be useful throughout the test lifecycle – also known as specification based testing – Applies for Functional and Non-Functional testing Input Output If Output = Expected result then pass

Black and White Box Testing Definition of White Box Testing white-box testing: Testing based on an analysis of the internal structure of the component or system.

Black and White Box Testing What is White Box Testing? – testing based upon the structure of the code – typically undertaken at Component and Component Integration Test phases by development teams – also known as structural or glass box testing or structure based testing Input Output

Dynamic black-box testing • Dynamic black-box testing is testing without having an insight into the details of the underlying code. – Dynamic, because the program is running – Black-box, because testing is done without knowledge of how the program is implemented. • Sometimes referred to as behavioral testing. • Requires an executable program and a specification (or at least a user manual). • Test cases are formulated as a set of pairs – E. g. , (input, expected output)

Test Data and Test Cases • Test data: Inputs which have been devised to test the system. • Test cases: Inputs to test the system and the predicted outputs from these inputs if the system operates according to its specification.

Test-to-pass and test-to-fail • Test-to-pass: – assures that the software minimally works, – does not push the capabilities of the software, – applies simple and straightforward test cases, – does not try to “break” the program. • Test-to-fail: – designing and running test cases with the sole purpose of breaking the software. – strategically chosen test cases to probe for common weaknesses in the software.

Black-box testing • Characteristics of Black-box testing: – Program is treated as a black box. – Implementation details do not matter. – Requires an end-user perspective. – Criteria are not precise. – Test planning can begin early.

Black, White and Experienced based Black (Specification Based) White (Structure Based) Experience (Black box) • Based on requirements • From the requirements, tests are created • Specification Models can be used for systematic test case design Techniques • Equivalence Partitioning • Boundary Value Analysis • Decision Tables • State Transition Testing • Use Case Testing • Based on code and the design of the system • The tests provide the ability to derive the extent of coverage of the whole application Techniques • Statement coverage • Branch Coverage • Decision Coverage • Based on the knowledge of the tester • Using past experienced use & intuition to “guess” where errors may occur Techniques • Error Guessing • Exploratory Testing

5. 3 Specification based or Black Box Techniques Equivalence Partitioning Boundary Value Analysis Decision Table Testing State Transition Testing Use Case Testing Other black box test techniques

Black Box Test Techniques 5. 3. 1 Equivalence Partitioning • aim is to treat groups of inputs as equivalent and to select one representative input to test them all • Best shown in the following example…. – If we wanted to test the following IF statement: – ‘IF VALUE is between 1 and 100 (inclusive) (e. g. VALUE >=1 and VALUE <= 100) THEN …. . ’ – We could put a range of numbers as shown in the next slide through test cases

Black Box Test Techniques Equivalence Partitioning IF Value >= 1 AND Value <= 100 THEN …. 101 0 37 1 -1 OUT OF RANGE 19 65 48 53 IN RANGE 99 87 1000 OUT OF RANGE

Black Box Test Techniques Equivalence Partitioning ( EP) • the numbers fall into a partition where each would have the same, or equivalent, result i. e. an Equivalence Partition (EP) or Equivalence Class • EP says that by testing just one value we have tested the partition (typically a mid-point value is used). It assumes that: 1. if one value finds a bug, the others probably will too 2. if one doesn't find a bug, the others probably won't either

Black Box Test Techniques Equivalence Partitioning • in EP we must identify Valid Equivalence partitions and Invalid Equivalence partitions where applicable (typically in range tests) • the Valid partition is bounded by the values 1 and 100 • plus there are 2 Invalid partitions

Black Box Test Techniques Equivalence Partitioning IF Value >= 1 AND Value <= 100 THEN …. 0 37 1 -1 19 Happy Path ‘INVALID’ PARTITION 65 48 53 99 87 101 1000 ‘VALID’ PARTITION Negative Scenario ‘INVALID’ PARTITION

Black Box Test Techniques Equivalence Partitioning • Time would be wasted by specifying test cases that covered a range of values within each of the three partitions, unless the code was designed in an unusual way • There are more effective techniques that can be used to find bugs in such circumstances (such as code inspection) • EP can help reduce the number of tests from a list of all possible inputs to a minimum set that would still test each partition

Black Box Test Techniques Equivalence Partitioning • If the tester chooses the right partitions, the testing will be accurate and efficient • If the tester mistakenly thinks of two partitions as equivalent and they are not, a test situation will be missed • Or on the other hand, if the tester thinks two objects are different and they are not, the tests will be redundant • EP can be used for all Levels of Testing • EP is used to achieve good input and output coverage, knowing exhaustive testing is often impossible • It can be applied to human input, input via interfaces to a system, or interface parameters in integration testing

Black Box Test Techniques 5. 3. 2 Boundary Value Analysis • Boundary Value Analysis (BVA) uses the same analysis of partitions as EP and is usually used in conjunction with EP in test case design • As with EP, it can be used for all Test levels • BVA operates on the basis that experience shows us that errors are most likely to exist at the boundaries between partitions and in doing so incorporates a degree of negative testing into the test design • BVA Test cases are designed to exercise the software on and at either side of boundary values

Black Box Test Techniques Boundary Value Analysis For our previous example of ‘Value >= 1 and Value <= 100’ ( 6 values as test cases are tested ) Value = 1 (valid) Value = 0 (invalid) Value = 100 (valid) Value = 2 (valid) Value = 99 (valid) Value = 101 (invalid)

Black Box Test Techniques Boundary Value Analysis • find the boundary and then test one value above and below it • ALWAYS results in two test cases per boundary for valid inputs and three tests cases per boundary for all inputs ( 2 valid and 1 invalid) • inputs should be in the smallest significant values for the boundary (e. g. Boundary of ‘a > 10. 0’ should result in test values of 10. 0, 10. 1 & 10. 2) • only applicable for numeric (and date) fields

Black Box Test Techniques 5. 3. 3 Decision Table Testing • Table based technique where – Inputs to the system are recorded – Outputs to the system are defined • Inputs are usually defined in terms of actions which are Boolean (true or false) • Outputs are recorded against each unique combination of inputs • Using the Decision Table the relationships between the inputs and the possible outputs are mapped together • As with State Transition testing, an excellent tool to capture certain types of system requirements and to document internal system design. As such can be used for a number of test levels • Especially useful for complex business rules

Black Box Test Techniques Decision Table Structure Inputs / Actions Output / Response Test 1 Test 2 Test 3 Input 1 T T F Input 2 T T F Input 3 T DON’T CARE F Input 4 T F T Response 1 Y Y N Response 2 Y N Y Response 3 N Y N Each column of the table corresponds to a business rule that defines a unique combination of conditions that result in the execution of the actions associated with that rule The strength of Decision Table testing is that it creates combinations of conditions that might not otherwise have been exercised during testing

Black Box Test Techniques Decision Table Example Test 1 Test 2 Test 3 > 55 yrs old F T T Smoker F T F Exercises 3 times a week + T F T History of Heart Attacks F T F Insure Y N Y Offer 10% Discount N N Y Offer 30% Discount Y N N What will be the out come of the following business rules? Joe is a 22 year old non smoker who goes to the gym 4 times / week and has no history of heart attacks in his family ( first business rule : test 1) Kevin is 62 year old non smoker who swims twice a week and plays tennis. He has no history of heart attacks in his family (third business rule: Test 3)

Black Box Test Techniques 5. 3. 4 State Transition Testing • State Transition Testing uses the following terms: 1. state diagram: A diagram that depicts the states that a component or system can assume, and shows the events or circumstances that cause and/or result from a change from one state to another. [IEEE 610] 2. state table: A grid showing the resulting transitions for each state combined with each possible event, showing both valid and invalid transitions. 3. state transition: A transition between two states of a component or system. 4. state transition testing: A black box test design technique in which test cases are designed to execute valid and invalid state transitions. Also known as N-switch testing. • An excellent tool to capture certain types of system requirements and to document internal system design. As such can be used for a number of test levels • Often used in testing: – Screen dialogues – Web site transitions

Black Box Test Techniques State Transition Diagram Starting State A Event/Action etc State B Transition Between start and end states Event from outside the system Action triggered by Event End State

Black Box Test Techniques State Transition Example Simplified Car Gears Change Down/ Move Back Reverse Neutral Change Up/ Accelerate Change Down/ Decelerate 1 st Gear Change Up/ Accelerate Change Down/ Decelerate 2 nd Gear Change Up/ Accelerate Change Down/ Decelerate 3 rd Gear Change Up/ Accelerate

Black Box Test Techniques State Transition - Switch Coverage • Switch Coverage is a method of determining the number tests based on the number “hops” between transitions. Sometimes known as Chow 0 -Switch Coverage (1 hop) R N N R N 1 1 N 1 2 2 1 2 3 3 2 1 -Switch Coverage (2 hops) R N 1 R N R N N 1 2 N 1 N R Etc. These hops can be used to determine the VALID tests

Black Box Test Techniques State Transition - State Table • While Switch testing helps determine the valid tests, we also need to look for the invalid tests. Change Up Change Down R Acc / Neutral Null N Acc/1 st Gear Dec / Reverse 1 st Acc/ 2 nd Gear Dec / Neutral 2 nd Acc / 3 rd gear Dec / 1 st Gear 3 rd Null Dec / 2 nd Gear Invalid tests are those identified by a null output in this case • Changing down from reverse • Changing up from 3 rd

Black Box Test Techniques State Transition – Another example for a Theatre Show reservation Request Show Options 1 2 Choose Show Options provided Reserve Show Reservation Made Show selected Change Mind/ 3 2' Cancel reservation Return to Options Pay for Show 4 4' Show Reservation Paid For 5' Cancelled Reservation 5 Cancel reservation/ Issue Refund Issue Ticket 6' Cancel reservation (return ticket)/Issue Refund Ticket Received 6

Black Box Test Techniques State Transition – Another type of State Table Current State Event/ Next State Event/ Next State A B C D E F 1 3 SS 1 1 2 2 3 4 4 ES ES SS A B 1 2 C D 3 E 4 F ES

Black Box Test Techniques 5. 3. 4 Use Case Testing • Uses Cases are method of describing requirements • Structured approach to requirements design • Usually has one flow (though may also have alternate flows) • Each use case is a process flow or scenario • Particularly useful in determining tests for (Functional) Systems Test and for UAT • Also good at detecting defects in the integration of interfaces • The main parts of a Use Case are: – Actors Users of the systems – Pre conditions What are the starting requirements for the use case – Post conditions The state the system will end up in once completed

Use Case Example ( as requirements representation or text which are the specific business scenarios) Tester can extract 6 test cases as positive scenarios. Negative scenarios are application dependent Precondition: customer must have an account Postcondition: due to transactions, balance increased or decreased

Other Black Box Test Techniques • Syntax testing ( critical item testing ) — test cases are prepared to exercise the rule governing the format of data in a system (e. g. a Zip or Postal Code, a telephone number) • Random testing ( using random generator) — test cases are selected, possibly using a pseudorandom generation algorithm, to match an operational profile Note: mainly Black Box Test Techniques are more commonly used than the white Box Test Techniques

5. 4 Structure Based or White Box Test Techniques ( source code is available for tester ) Statement Testing Decision Testing Assessing Completeness (Coverage) Other White Box test techniques • The next examples that will be shown are quite simplistic and, as such, are possible to determine manually. • However, in a real life scenario, the complexity and scale of code means that this would normally be undertaken with the assistance of a test tool ( automated)

White Box Test Techniques 5. 4. 1 Statement Testing – aim is to display that all executable statements have been run at least once – coverage measurement is: number of statements executed total number of statements

White Box Test Techniques Statement Testing Example 1 1 Control Flow 2 1. Read vehicle Graph 3 2. Read colour 4 3. If vehicle = ‘Car’ Then 4. If colour = ‘Red’ Then 5. 6. 5 Print “Fast” 6 End If 7

White Box Test Techniques Statement Testing 1 2 Example 2 1. Read A 2. If A > 40 Then 3. A=A*2 4. End If 3 4 5 5. If A > 100 Then 6. 6 A = A – 10 7. End If 7

White Box Test Techniques Example 3 Statement Testing 1. Read bread 1 2 2. Read filling 3/9 3. If bread = ‘Roll’ Then 4. If filling = ‘Tuna’ Then 5. 6. Price = 1. 50 Else 7. 8. 4/6 7 Price = 1. 00 10 8 End If 9. Else 10. Price = 0. 75 11. End If 5 11

White Box Test Techniques 5. 4. 2 Decision Testing – Aim is to demonstrate that all Decisions have been run at least once – Coverage measurement is: the number of Decisions executed the total number of Decisions – Note that Decision testing is related to Branch testing – Decisions are lines of code where there are two or more alternative routes based on a decision that has to be made, i. e. IF/THEN/ELSE – Branches are where there are one of two or more alternative routes from a line of code but also include JUMPs and GO TOs

White Box Test Techniques Decision Testing Example 1 1 2 1. Read vehicle 3 2. Read colour 4 3. If vehicle = ‘Car’ Then 4. 5. 6. 5 If colour = ‘Red’ Then Print “Fast” 6 End If 7

White Box Test Techniques Decision Testing Example 2 1. Read A 3 2. If A > 40 Then 3. A=A*2 4. End If 4 5 5. If A > 100 Then 6. 6 A = A – 10 7. End If 7

White Box Test Techniques Decision Testing Example 3 1 1. Read bread 2 2. Read filling 3/9 3. If bread = ‘Roll’ Then 4. If filling = ‘Tuna’ Then 5. 6. Price = 1. 50 Else 7. 8. 4/6 Price = 1. 00 End If 7 10 8 9. Else 10. Price = 0. 75 11. End If 5 11

Comments • Note that in this Technique, the Ifs and associated Elses are wrapped up as One Node. • The final example requires three test cases to execute all of the possible decision branches from the two branch points (3/9 & 4/6). The left hand side would be exercised by an input of “Cheese Sandwich”, the centre line by “Cheese Roll” and the right hand side by “Tuna Roll”. Corresponding outputs would be 0. 75, 1. 00 and 1. 50. • For this example, the number of test cases is the same for Branch / Decision testing. • ‘While a single test case is sufficient to test every line of code in this module (for example, use a=6 and b=3 as input), it is apparent that this level of coverage will miss testing many paths. Thus, statement coverage, while a beginning, is generally not an acceptable level of testing.

White Box Test Techniques Statement and Decision Testing The pseudo-code can be expressed in Flow-chart format 1. Read A 2. If A > 40 Then 3. A=A*2 4. End If 5. If A > 100 Then 6. A = A – 10 7. End If What path is required to provide 100% Statement coverage? Answer : a, b, c, d, g

White Box Test Techniques 5. 4. 3 Assessing Completeness (Coverage) 1. Read bread We already know: ( see next slide ) 2. Read filling 3. If bread = ‘Roll’ Then Statement Coverage = 3 (#5, 7, 10) Number of Statements = 7 the number of If filling = ‘Tuna’ Then Statements covers ALL EXECUTABLE Statements, not just the ‘perform an Price = 1. 50 action’ ones. So we include our IFs and WHILEs but exclude any ELSEs and Else ENDIFs (#all except 6, 8, 9, 11) 4. 5. 6. 7. Price = 1. 00 8. End If 9. Else 10. Price = 0. 75 11. End If For Global Assessment Number of Branches = 4 (T and F of each IF) (#3 and #4)

Assessing Completeness (Coverage)-contn 1 Read Bread 2 N 9 Number of Branches = 4 = Y&N of Node (3) + Y&N of Node (4) Read Filling If 3 bread = ‘Roll’ Statement Coverage = 3 = Node (5) + Node (7) + Node (10) Y If 4 Filling = ‘Tuna’ N 6 7 10 Print 1. 00 EGP Print 0. 75 EGP 5 Print 1. 5 EGP 8 END If 11 END If Y

White Box Test Techniques Assessing Completeness (Coverage) 1. Read bread For Partial Assessment 2. Read filling • Based on the following partial test set: 3. If bread = ‘Roll’ Then Bread Filling 4. Roll Tuna Sandwich Ham If filling = ‘Tuna’ Then 5. 6. Price = 1. 50 Else 7. 8. Price = 1. 00 End If 9. Else 10. Price = 0. 75 11. End If • What is the test Statement Coverage and Decision Coverage? ( for the two cases together ) • Therefore Statement Coverage = 6/7 (85%) • And Decision Coverage = 3/4 (75%) See next slide for details

Comments • In this example we know that there are 7 EXECUTABLE statements (1, 2, 3, 4, 5, 7, and 10) and there are 2 decision points that each require 2 test paths – therefore 4 paths. • With the test set of “Tuna Roll” and “Ham Sandwich” how complete is our statement test coverage and how complete is our decision test coverage? • “Roll Tuna” tests 3 executable statements (1, 2, 3, 4, 5) • “Sandwich Ham” tests 3 executable statements (1, 2, 3, 10) • Using both will test 6 unique executable statements (all except line 7) • Since there are 7 executable statements, this test set only tests 85% (6/7) of the executable statements. This is the test statement coverage. (Executable statement 7 is not tested). • • “Roll Tuna” tests 2 decisions points (3 -right and 4 -right) “Sandwich Ham” tests 1 decision point (3 -left) Using both will test 3 unique decisions (3 -right, 3 -left and 4 -right) Since there are 4 paths, this test set only tests 75% (3/4) of the decisions. (Decision 4 -left is not tested).

Choosing test Techniques • How do you choose the right technique? – Type of system – Standards – Customer or contractual requirements – Level of risk – Type of risk – Testing objectives – Documentation available – Knowledge / skills of the testers – Time and budget – Development processes • Pick the right techniques for the right situation