A New Software Testing Approach Based on Domain

A New Software Testing Approach Based on Domain Analysis of Specifications and Programs Ruilian Zhao Michael R. Lyu Yinghua Min Computer Science Dept. Beijing University of Chemical Technology Computer Science Dept. Chinese University of Institute of Computing Tech. Chinese Academy of Sciences in Beijing 2020/9/10 Hong Kong 1

Outline Introduction The domain testing approach Program operational domain Example study Conclusion 2

Introduction Software testing can usually be classified into two categories: functional testing and structural testing. An obvious problem is that program control structures or software specifications themselves may be incorrect. Consequently, if testers have some knowledge about the structure as well as the specification of the program under test, better test effectiveness can be achieved. 3

Introduction A key issue in software testing is how to select test data from program input domain. There a large number of test data selection strategies, such as equivalence partitioning, path testing, boundary value analysis, domain testing and so on. All of these strategies are based on partitioning input domain, referred to as partition testing. 4

Introduction However, we will show with an example that partition testing strategies are relatively ineffective in detecting faults having to do with small shifts in input domain boundary. 5

Software specification code implementation Introduction functional domain operational domain Coincidence? If the two domains are not coincided with each other exactly, some software faults may be located. 6

Introduction we present a new software testing approach, referred to as a domain analysis testing approach based on specifications and programs. Test cases are generated from coincidence verification of functional domain and operational domain of the program under test. 7

Introduction In order to obtain operational domain of a program, we have developed an automated determination system of operational domain, called ADSOD. A preliminary evaluation on fault detection effectiveness and code coverage illustrates that the domain analysis testing approach is highly effective in detecting faults due to small shifts in the input domain boundary, and is more economical in test case generation than the partition testing strategies. 8

The domain testing approach 1. The principle of boundary test case selection By the domain D of a program, we mean a cross product : where xi (i=1, 2, …, n) is an input variable of the program. The projection of domain D on xi axis, represented by D xi, is a subset of the one dimension space. It satisfies 9

The domain testing approach 1. The principle of boundary test case selection Definition 1: Reflecting domain of projection. By reflecting domain R(D, xi=a) of projection of domain D on xi axis, we mean in n-1 dimension space where xi D xi. 10

The domain testing approach 1. The principle of boundary test case selection We define the following reflecting domains in succession: … where are constants 11

The domain testing approach 1. The principle of boundary test case selection Theorem 1: Let D be an n-dimension bounded closed convex domain. Let be defined as above. Then. B(D) denote the set of boundary points of domain D. 12

The domain testing approach 1. The principle of boundary test case selection 13

The domain testing approach 1. The principle of boundary test case selection By above successive definition, we have that is a one dimension subset of the original domain D, that is, We denote it by . 14

The domain testing approach 1. The principle of boundary test case selection Definition 2: The set of boundary test cases T(D). Let be a center point of projection of domain on axis , respectively. The set of boundary test cases of domain D is defined as follows: 15

The domain testing approach The following claim describes how to select boundary test cases to verify the coincidence of the two domains. Claim: Suppose functional domain and operational domain are bounded closed convex. Then, the sufficient and necessary condition that the two domain coincides with domain is: 1). ; 2). For point in any neighborhood of point q, or then, either or. 16

The domain testing approach In order to check the coincidence of functional domain and operational domain , not only the sets of boundary test cases T(Df) and T(Dp) need to be examined, but also some test cases in neighborhoods of the boundary points should be taken into consideration by above claim. 17

The domain testing approach 2. Test data generation based on the domain analysis The test data generation based on the domain analysis consists of the following steps: Find out all input variables from both the specification and the program under test. 1) 2) Derive That is, for all a): compute b): check whether and . is true or false. 18

The domain testing approach 2. Test data generation based on the domain analysis 3) Develop test case according to or . Each or has 2 boundary points. Test cases take three values for each boundary point, that is, the boundary point itself, boundary point + , and boundary point - , where is a small amount. So, 2*3 test cases are produced for each input variable xi. 19

The domain testing approach As a result, the testing approach only needs 2*3*n test cases, where n is the number of input variables. However, in boundary-interior path testing, 6 L test cases are required in the worst case. In domain testing, 2*N*M test points are required. It is obvious that the domain analysis testing approach enjoys the advantage of fewer test cases than path testing and domain testing, and the test generation is independent on path selection 20

Program operational domain The determination of program operational domain is a foundation for the domain analysis testing approach. We develop an automated determination system of operational domain, called ADSOD, whose input is a program written in C programming language, and the output is its operational domain. 21

Program operational domain Firstly, the pro-processor is derived to create the various tables and to construct the procedures based_on_expression_er() for specific expressions. Secondly, the procedures are complied together with original program P to generate executable codes. Then, the domain Dxi is computed by invoking the module domain_determination(). 22

Program operational domain A specific expression e is the one that the variables in e are related to input variable and e can not take some values due to some constraints, such as when e is used as a divisor or as a parameter of some standard subroutines In order to identify illegal values of input variables in a specific expression e, we construct a procedure, based_on_expression_er(), with respect to the expression e by using program slicing technique. 23

Program operational domain The procedure, based_on_expression_er(), is made up of a slice and some instrumentation statements. The illegal values of input variables x for the expression e can be identified by executing the procedure. 24

Program operational domain To verify the validity of the ADSOD system, a number of different kinds of practical programs have been executed. Experimental results illustrate that ADSOD supports not only the determination of input domain of integer and real data types, but also non-numeric data types such as characters string types. 25

Example study We compare the effectiveness of our domain analysis testing approach with partition testing strategies in fault detection with an example Max. Min, which is a variation of the program taken from reference[14] There are 30 feasible paths in the Max. Min program according to its control flow graph. 30 test cases are generated as to these feasible paths, denoted by test_set_1= , and Max. Min program are executed with these test cases. 26

Example study These test cases reveal three software faults in Max. Min: Fault 1). Line 14: Checking option “-ceiling” should be put down before the line 12. Fault 2). Line 16: The exit of error should be added when arg[1] is not given. Fault 3). Line 20: The exit of error should be added when arg[1] is not given. 27

Example study The simplified domain testing strategy is applied to develop ON-OFF test points for each border of each path domain. In the worse case, there are 30+312=342 ON-OFF test points. However, they only detect the same three software faults. 28

Example study The operational domain of Max. Min program can be obtained by calling the ADSOD system. The functional domain is specified by its specification. Since the specification does not specifies the values that input variables can hold. The specification should be described in more detailed, such that. 29

Example study Test requirements can be produced according to T(Dp) On the basis of these test requirements, 15 test cases are developed, test_set_2= and Max. Min program are run with these test cases. 30

Example study The test cases expose five software faults. They are: Fault 4). Line 16: The BUFSIZE should be BUFSIZE – 1. Fault 5). Line 20: The BUFSIZE should be BUFSIZE – 1. Fault 6). Line 38: The BUFSIZE should be given. Fault 7). The value of arguments should be between – 9223372036854775808 and 9223372036854775807. Fault 8). Line 14: Checking option ‘-ceiling’ should be put down before the line 12. In these faults, only fault 8) is same as the fault 1). 31

Example study We measure the coverage of test_set_1, test_set_2 and test_set_1 + test_set_2 using ATAC This demonstrates that two test sets with identical coverage could have very different fault detection effectiveness. 32

Conclusion This domain analysis testing approach is different from boundary value analysis. Preliminary experimental results indicate that the new testing approach is highly effective in detecting the faults related to small shifts in input domain boundary, and is more economical in test case generation than the partition testing strategies. 33

34