Introduction to Software Testing Black Box Testing L

Introduction to Software Testing Black Box Testing L 10 2020

Verification & validation & levels of software testing

Static Testing Dynamic Testing done without executing the program Testing done by executing the program This testing does verification process Dynamic testing does validation process Static testing is about prevention of defects Dynamic testing is about finding and fixing the defects Static testing gives assessment of code and documentation Dynamic testing gives bugs/bottlenecks in the software system. Static testing involves checklist and process to be followed Dynamic testing involves test cases for execution This testing can be performed before compilation Dynamic testing is performed after compilation Static testing covers the structural and statement coverage testing Dynamic testing covers the executable file of the code Cost of finding defects and fixing is less Cost of finding and fixing defects is high Return on investment will be high as this process involved at early stage Return on investment will be low as this process involves after the development phase More reviews comments are highly recommended for good quality More defects are highly recommended for good quality. Requires loads of meetings Comparatively requires lesser meetings

Types of software testing (verification & validation) Verification tests are also known as static tests as no execution of artifact is required to run it during these tests. Verification tests include requirement specification reviews, design reviews, source code reviews etc. Validation tests are also known as dynamic tests as execution of artifact under test is required to run it during these tests. Validation tests are further divided into white box and black box tests. Verification and Validation are two measures used to check that the software product meets the requirements specifications. Together they help improve software quality.

Types of software testing (verification & validation) White box tests are named so as the source code is used during software testing. White box tests are also known as structural tests as they test small parts (structures) of the software product. Unit tests and integration tests fall under these tests. Black box tests are named so as the source code is not used during software testing. Instead the executable binary machine code is used. Black box tests are further divided into functional and non functional tests. Non functional tests are further divided into performance, security, usability etc. tests. Black box tests are done both at system level as well as at deployment (user acceptance) level. Regression tests are done for both white box as well as black box tests.

Verification & validation and associated software testing types

Verification & Validation Verification is that part of software testing where the testing is done on any software project artifact which has no executable part available. Software requirement specifications, software designs, source code (testing to view only the source code and not running it) are project artifacts which are tested using verification techniques. Requirement specifications are tested using requirement specification reviews, software designs are tested using design reviews and source code (construction) is tested using source code reviews.

Verification & Validation is that part of software testing where the testing is done on software product by executing its source code. Business logic at class (component) level is tested using unit testing, integration of component is tested using integration testing, the software product is tested using system testing and finally the deployment of the software product is tested by end users using user acceptance testing.

VELIDATION • Black box testing 2/27/2021 Black Box Testing Slide : 9

What is Black Box Testing? • Black box testing, which is also known as behavioral, opaque-box, closed-box, specification-based or eye-to-eye testing, is a Software Testing method that analyses the functionality of a software/application without knowing much about the internal structure/design of the item that is being tested and compares the input value with the output value. 2/27/2021 Black Box Testing Slide : 10

• The main focus in black box testing is on the functionality of the system as a whole. The term ‘behavioral testing’ is also used for black box testing. Behavioral test design is slightly different from the black-box test design because the use of internal knowledge isn’t strictly forbidden, but it’s still discouraged. • Each testing method has its own advantages and disadvantages. There are some bugs that cannot be found using the only black box or only white box technique. Majority of the applications are tested by black box method. We need to cover the majority of test cases so that most of the bugs will get discovered by a black-box method. • This testing occurs throughout the software development and Testing life cycle i. e in Unit, Integration, System, Acceptance, and regression testing stages. • Can be functional and non functional 2/27/2021 Black Box Testing Slide : 11

What is Black Box Testing? • Black box testing is done without the knowledge of the internals of the system under test. 2/27/2021 Black Box Testing Slide : 12

Characteristics of Black Box Testing • Done based on requirements • Addresses (should address) stated as well as implied requirements • Encompasses the end-user perspective • Checks for valid and invalid conditions / inputs • May or may not know the technology aspects of the product 2/27/2021 Black Box Testing Slide : 13

Differences Between White Box and Black Box Testing White Box Testing Has no access to program Has access to program code 2/27/2021 Requires external perspective Requires knowledge of program code Set of techniques applicable to all other phases of testing Typically applies only to unit testing, where code is involved Black Box Testing Slide : 14

Principles of Black Box Testing • It is not possible to exhaustively test a product • Choose tests so that we can • Test “as much of the external functionality as possible” • Uncover “as many defects as possible” • Use “as short a time as possible” Have methodologies that choose tests that have a higher likelihood of uncovering new defects. 2/27/2021 Black Box Testing Slide : 15

2/27/2021 Black Box Testing Slide : 16

2/27/2021 Black Box Testing Slide : 17

Techniques / Methodologies of Black Box Testing • Requirements-based testing • Positive and negative testing • Boundary value analysis • Decision tables • Equivalence partitioning • State-based testing • Compatibility testing • User documentation testing • Domain testing (leads to ad hoc testing) 2/27/2021 Black Box Testing Slide : 18

General Format of Discussion of Techniques • Present some reasoning where applicable. • List out one or two examples. • Walk through the examples. • Summarize the process for using the technique. 2/27/2021 Black Box Testing Slide : 19

Requirements-Based Testing • Done to ensure that all requirements in SRS are tested • Difference between implicit and explicit requirements • Review requirements first to ensure they are consistent, correct, complete and testable • Review enables translation of (some of) the implied requirements to stated requirements • A reviewed SRS tabulates requirements, along with a requirements id and a priority • This is the genesis of a Requirements Traceability Matrix 2/27/2021 Black Box Testing Slide : 20

2/27/2021 Black Box Testing Slide : 21

2/27/2021 Black Box Testing Slide : 22

Zooming-In on the Testing Part of RTM • Table 4. 2 • Test condition: Different ways of testing a requirement (different types of mapping) • Test case: Different conditions / scenarios for a given requirement • Phase of testing – helps scheduling • Importance of documenting expected results 2/27/2021 Black Box Testing Slide : 23

Positive and Negative Testing • Positive testing to check that the product does what it is supposed to • Behaves correctly when given right inputs • Maps to a specific requirement • “Coverage” is defined better • Negative testing to show that the product does not fail when given unexpected inputs • Tries to break the system • No direct mapping to a specific requirement • “Coverage” more challenging • Table 4. 4 and 4. 5, Page 83 2/27/2021 Black Box Testing Slide : 24

2/27/2021 Black Box Testing Slide : 25

Boundary Value Analysis • Boundary testing is the process of testing between extreme ends or boundaries between partitions of the input values. • These extreme ends like Start- End, Lower- Upper, Maximum. Minimum, Just Inside-Just Outside values are called boundary values and the testing is called "boundary testing". • The basic idea in boundary value testing is to select input variable values at their: • • • 2/27/2021 Minimum Just above the minimum A nominal value Just below the maximum Maximum Black Box Testing Slide : 26

Boundary Value Analysis • Most defects come up near “boundaries” • Reasons from a white box perspective: • Programmers tentativeness in using the right relational operator (< or < = ? ) • Multiple ways of implementing loops • Confusing array subscripts • Reasons from a black box perspective: • Unclear requirements • Ambiguous or “it depends” mindset! 2/27/2021 Black Box Testing Slide : 27

Example 1: Volume Discount Calculations • Product pricing depends on the quantity procured • Based on pre-defined “slabs” 2/27/2021 Black Box Testing Slide : 28

2/27/2021 Black Box Testing Slide : 29

Test cases for input box accepting numbers between 1 and 1000 using Boundary value analysis: 1) Test cases with test data exactly as the input boundaries of input domain i. e. values 1 and 1000 in our case. 2) Test data with values just below the extreme edges of input domains i. e. values 0 and 999. 3) Test data with values just above the extreme edges of input domain i. e. values 2 and 1001. Boundary value analysis is often called as a part of stress and negative testing. 2/27/2021 Black Box Testing Slide : 30

Boundary Value Analysis (BVA): Summary of Process • Look for any kind of gradation or discontinuity in data values that affects computation – the discontinuities are the boundary values, requiring thorough testing. • Look for any internal limits like limits on resources (like the example of buffers used in database for caching). The behaviour of the product at these limits should also be the subject of boundary value testing. • Also include in the list of boundary values, documented limits on hardware resources. For example, if it is documented that a product will run with minimum 4 MB of RAM, make sure you include test cases for the minimum RAM (i. e. , 4 MB in this case). • The examples given above discuss boundary conditions for input data – the same analysis needs to be done for output variables also. 2/27/2021 Black Box Testing Slide : 31

Decision Tables • A program’s behaviour is characterized by several decision variables. • Each decision variable specifies a boolean condition. • The distinct combinations of these decision variables lead to different scenarios. • Each scenario occupies a row in the decision table, and the row also has expected results. • One representative data point from each scenario needs to be tested. 2/27/2021 Black Box Testing Slide : 32

Decision Table example of business requirement for a login screen. 2/27/2021 Black Box Testing Slide : 33

The first step we do is to name all the branches and leave with numbers or alphabets as below. 1, 2, 3 are the leaves and a, b & c are the branches. Then, we have to create a decision table as shown below: Fig 1. 1 Decision table for business flow 2/27/2021 Black Box Testing Slide : 34

Points to Remember • All the validations specified in the decision boxes should be made of the columns on the table. • All the results (leaves) mentioned in the flow diagram should be covered in the decision table. • All combinations of inputs needed to obtain a certain result shall be mentioned in the combinations column and can be included while writing the test cases. • After completing the decision table, one has to just verify whether all the branches and leaves in the logical tree are covered. 2/27/2021 Black Box Testing Slide : 35

Advantages of Using Decision Table Technique #1) Any complex business flow represented as a diagram can be easily covered in this technique. #2) It provides quick confidence on the test cases. One need not have to review his own test cases multiple times to gain confidence. #3) Easy to understand. Anyone can make test cases from this decision table template. #4) Rework on the test cases and test scenario can be totally avoided, as it gives complete coverage at the first shot. 2/27/2021 Black Box Testing Slide : 36

Summary of Decision Table • • • 2/27/2021 Identify the decision variables. Identify the possible values of each of the decision variables. Enumerate the combinations of the allowed values of each of the variables. Identify the cases in which values assumed by a variable (or by sets of variables) are immaterial for a given combination of other input variables. Represent such variables by the Don’t Care symbol. For each combination of values of decision variables (appropriately minimized with the Don’t Care scenarios), list out the action or expected result. Form a table, listing on each but the last column one decision variable. On the last column, list the action item for the combination of variables in that row (including Don’t Cares, as appropriate) Black Box Testing Slide : 37

Equivalence Partitioning (EVP) • Generalization of BVA / decision table • Divide the (potentially infinite) set of values into a set of equivalence classes or partitions • One element of the class can act as a representative for the entire class • Results of the test for that one element is extrapolated to all the other elements 2/27/2021 Black Box Testing Slide : 38

Example of EVP 2/27/2021 Black Box Testing Slide : 39

2/27/2021 Black Box Testing Slide : 40

Summary of Process for EVP • Choose criteria for doing the equivalence partitioning (range, list of values, etc. ). • Identify the valid equivalence classes based on the above criteria (number of ranges, allowed values, etc. ). • Select a sample data from that partition. • Write the expected result based on the requirements given. • Identify special values, if any, and include them in the table. • Check to have expected results for all the cases prepared • If the expected result is not clear for any particular test case, mark appropriately and escalate for corrective actions. If you cannot answer a question, or find an inappropriate answer, consider whether you want to record this issue on your log. 2/27/2021 Black Box Testing Slide : 41

State- or Graph-Based Testing • Useful for • Language processors or compilers • Workflow modelling • Dataflow modelling 2/27/2021 Black Box Testing Slide : 42

Example of Graph-Based Testing for an SQL Command SELECT * 2 e> 1 <c ol na m FROM 3 , 4 5 <table name> 6 <table name> <col name> , FROM WHERE 7 2/27/2021 Black Box Testing Slide : 43 …

2/27/2021 Black Box Testing Slide : 44

Process for Language processors Testing Using Graph Method • • • 2/27/2021 Identify the grammar for the scenario. In the above example, we have represented the diagram as a state machine. In some cases, the scenario can be a contextfree grammar, which may require a more sophisticated representation of a “state diagram”. Design test cases corresponding to each valid state-input combination. Design test cases corresponding to the most common invalid combinations of state-input. Black Box Testing Slide : 45

Approval Approving Authority Requestor Finance y Buy & Expense (Transaction: Expense Report) Received Item 2/27/2021 m Ite Payment Available Off the shelf De Delivery Note er il v Budget Update & Purchase Order Requisition Vendor Black Box Testing Slide : 46

Workflow Modelling 2/27/2021 Black Box Testing Slide : 47

General Characteristics of Graph-Based Testing • • • 2/27/2021 The application can be characterized by a set of states. The data values (screens, mouse clicks, etc. ) that cause the transition from one state to another are well understood. The methods of processing within each state to process the input received are also well understood. Black Box Testing Slide : 48

Compatibility Testing • To ensure that the product works consistently with infrastructure components • Could be parameters of hardware, OS, network… • Concept of a compatibility matrix (Table 4. 11, Page 98) • Backward compatibility and forward compatibility 2/27/2021 Black Box Testing Slide : 49

User Documentation Testing • To check if what is stated in the document is available in the product • To check if what is there in the product is available in the document • Documents include user guides, installation guides, set-up guides, read-me files, software release notes and on-line help 2/27/2021 Black Box Testing Slide : 50

2/27/2021 Black Box Testing Slide : 51

Benefits of Documentation Testing • User documentation testing aids in highlighting problems that have been overlooked during reviews. • High-quality user documentation ensures the consistency of documentation and product, thus minimizing possible defects reported by customers. • Results in less difficult support calls. • Contributes to better customer satisfaction and better morale of support staff. • New programmers and testers who join a project group can use the documentation to learn the external functionality of the product. • Customers need less training and can proceed more quickly to advanced training and product usage. • Reduced training costs for customers. 2/27/2021 Black Box Testing Slide : 52

Domain Testing 2/27/2021 Black Box Testing Slide : 53

Domain Testing • Have “domain people” perform tests by using the software • Business flow determines the test, not “logic” or “steps” • Tests what the users do on a typical business day • Captures the typical problems faced by users (not necessarily captured in SRS) 2/27/2021 Black Box Testing Slide : 54

When to Use What… When you want to test scenarios that have… The most effective black box testing technique is likely to be… Output values are dictated by certain conditions depending upon values of input variables Decision Tables Input values being in ranges, with each range exhibiting a particular functionality Boundary Value Analysis Input values being divided into classes (like ranges, list of values, etc), with each class exhibiting a particular functionality Equivalence Partitioning Positive and negative testing 2/27/2021 Workflows, process flows or language processors Graph based testing To ensure that requirements are tested and met properly Requirements Based Testing To test using the domain expertise rather than the product specification Domain Testing To ensure that the documentation is consistent with the product Documentation Testing Black Box Testing Slide : 55

Black Box Testing: Objectives (Recap) To understand: • What black box testing is • The differences between black box testing and white box testing • Different techniques of black box testing 2/27/2021 Black Box Testing Slide : 56

• https: //www. softwaretestinghelp. com/black-box-testing/ 2/27/2021 Black Box Testing Slide : 57