Aug 2009 Exterimental Study of ART Reliability An

Aug. 2009 Exterimental Study of ART Reliability An Experimental Evaluation of the Reliability of Adaptive Random Testing Methods Hong Zhu Department of Computing and Electronics, Oxford Brookes University, Oxford OX 33 1 HX, UK Email: hzhu@brookes. ac. uk

Motivation v General problem: to evaluate and compare software testing methods? Aug. 2009 Ø How v Existing works Ø Comparison criteria § Fault detecting ability § Cost A great number of software testing methods have been proposed, but few have been widely used in practice. Ø Comparison methods § Formal analysis • Subsumption relation and other relations • Axiomatic assessment § Experimental and empirical studies Exterimental Study of ART Reliability 2

What is a good software testing method? v Goodenough Aug. 2009 Ø Validity: and Gerhart (1975): Fault detecting ability § for all software under test (SUT) there is a test set generated by the method that can detect faults if any. Reliable and stable Ø Reliability: § if one such test set detects a fault, then ideally any test set of the method will also detect the fault. J. B. Goodenough and S. L. Gerhart, Toward a theory of test data selection, IEEE TSE, Vol. SE_3, June 1975. Exterimental Study of ART Reliability 3

Proposed idea v Research Aug. 2009 Ø hypothesis It is inadequate if testing methods are only compared on fault detecting ability. They should also be compared on how reliable they are in the sense of stably producing high fault detecting rates. v Research questions: Are testing methods differ from each other in reliability? Ø If yes, what are the factors that affect their reliabilities? Ø How to compare testing methods’ reliability? Ø § How to measure? § How to do experiments? Ø What are the practical implications? Exterimental Study of ART Reliability 4

The approach v Investigation of some specific software testing methods to test the hypothesis: Aug. 2009 Ø random software testing (RT) Random testing have been regarded • as effective as systematic testing • more efficient than systematic methods Ø adaptive random software testing methods (ART) Adaptive random testing have been proposed to further improve the effectiveness and efficiency of random testing by evenly distribute the test cases in the input space. (T. Y. Chen, et al. , 2004, …) Exterimental Study of ART Reliability 5

Background: (1) Random testing Aug. 2009 v Random testing (RT) is a software testing method that selects or generates test cases through random sampling over the input domain of the SUT according to a given probability distribution. Ø Representative random testing Ø Non-representative random testing Exterimental Study of ART Reliability 6

Aug. 2009 Definitions Strictly speaking, it should not be called test set as elements are ordered. But, it is a convention in the literature of ART to call them test set. Let D be the input domain, i. e. the set of valid input data to the SUT. A test set T=<a 1, a 2, …, an>, n 0, on domain D is a finite sequence of elements in D, i. e. ai D, for all i=1, 2, …, n. The number n of elements in a test set is called the size of the test set. A SUT is tested on a test set T=<a 1, a 2, …, an> means that the software is executed on inputs a 1, a 2, …, an one by one with necessary initialization before each execution. When n=0, the test set is called empty test, which means the software is not executed at all. Without loss of generality, we assume that test sets considered in this paper are non-empty. Exterimental Study of ART Reliability 7

Aug. 2009 Definitions A test set T=<a 1, a 2, …, an> is a random test set if the elements are selected or generated at random independently according to a given probability distribution on the input domain D. If the same input is allowed to occur more than once in a test set, we say that the random test sets are with replacement; otherwise, the test sets are called without replacement. In the experiment reported here, the random test sets are with replacement. Exterimental Study of ART Reliability 8

Background: (2) Adaptive random testing v Basic idea: Aug. 2009 Ø To evenly spread the test cases in the input space so that it is more likely to find faults in the software. Ø First to generate random test cases, then to manipulate the random test cases to achieve the goal of even spread. Ø It is proposed and investigated by Prof. T. Y. Chen and his research group. A variety of such manipulations have been proposed and investigated. Exterimental Study of ART Reliability 9

Aug. 2009 Adaptive Random Testing 1: Mirror Let D 0, D 1, D 2, …, Dk (k>0) be a disjoint partition of the input domain D. The sub-domains D 0, …, Dk are of equal size and there is an one-to-one mapping mi from D 0 to Di, i=1, …, k. The subdomain D 0 is called the original sub-domain. The other subdomains are called the mirror sub-domains. The mappings mi are called the mirror functions. Let T 0 =<a 01, a 02, …, a 0 n> be a random test set on D 0. The mirror of the original test sets T 0 in a mirror sub-domain Di through mi is a test set Ti, written mi(T 0), defined as follows. mi(<a 01, a 02, …, a 0 n>)=<ai 1, ai 2, …, ain>, where mi(a 0 j ) = aij, i=1, …, k, j=1, …, n. The mirror test set of T 0 , written Mirror(T 0), is a test set on D that Mirror(T 0) = <a 01, a 11, a 21, …, ak 1, a 02, a 12, …, ak 2, …, a 0 n, a 1 n, …, akn >. Exterimental Study of ART Reliability 10

Aug. 2009 Adaptive Random Testing 2: Distance Let || x, y || be a distance measure defined on input domain D. It is extended to the distance between an element x and a set Y of elements as || x, Y|| = min(||x, y||, y Y). A test set T=<a 0, a 1, a 2, …, an> is maximally distanced if for all i=1, 2, …, n, || ai, {a 0, …, ai-1}|| || aj, {a 0, …, ai-1} ||, for all j > i. Let T be any given test set. The distance manipulation of T is a re-ordering of T’s elements in the sequence so that it is maximally distanced, written Distance(T). A distance measure on a domain D is a function || || from D 2 to real numbers in [0, ) such that for all x, y, z D, || x, x || = 0; || x, y|| = ||y, x || and ||x , y || ||x , z || + || z, y||. Exterimental Study of ART Reliability 11

Aug. 2009 Adaptive Random Testing 3: Filter Let || x , y || be a distance measure defined on the input domain D. Let T be any given non-empty test set. Test set T’ =<a 1, a 2, …, an> is said to be ordered according to the distance to T, if for all i=1, 2, . …, n-1, ||ai, T|| ||ai+1, T||. Let S and C be two given natural numbers such that S > C > 0, and T 0, T 1, …, Tk be a sequence of test sets of size S>1. Let U 0=T 0. Assume that Ui=<a 1, …, aki> and Ti+1=<c 1, …, c. S>. We define T’i+1=<b 1, …, b. S> be a test set obtained from Ti+1 by permutation its elements so that T’i+1 is ordered according to the distance to Ui. Then, we inductively define Ui+1=<a 1, …, aki, b 1, …, b. C>, for i=0, 1, …, k-1. The test set Uk is called the test set obtained by filtering C out of S test cases. Exterimental Study of ART Reliability 12

Notes on the definitions of ART techniques Aug. 2009 v Strictly speaking, the results of the above manipulations are not random test sets even if the test sets before manipulation are random. v These manipulations can be combined together to obtain more sophisticated adaptive random testing. It is defined here to demonstrate that testing methods For example, in mirror adaptive random testing, the input domain v The set of manipulations defined above may not isbe defined in the form of an algorithm can also be defined partitioned. AThere randomare test set is first generated on the original on subcomplete. also other manipulations formally as manipulations. domain. It is then manipulated by the Distance operation and then random testtheset; Mirrored into mirror sub-domain. of ART testing methods This simplifies the descriptions and formally, v Inconcisely the literature, ART methods are usually described in This enables us tocase recognise easily the various different the form of test generation algorithms rather than ways in which operators. they can be combined. manipulation v The filter manipulation is not used in the experiments reported in this paper. Exterimental Study of ART Reliability 13

The general framework of experiments with software testing method Si can be the same as or different from Sj. Aug. 2009 v v v Select a set of software under test (SUT): S 1, S 2, …, Sk, k>0; Select a measurement f of fault detecting ability on single test set; Applying the test method to generate test sets T 1, T 2, …, Tk Test the SUT on test sets and measure fault detecting ability: Ø Assume that m 1, m 2, …, mk are the fault detecting abilities of T 1, T 2, …, Tk as measured by f in testing SUTs S 1, S 2, …, Sk, Calculate the performance of the test method using a fault detecting ability measure Exterimental Study of ART Reliability 14

Aug. 2009 Background: (4) Measurements of fault detecting ability P-measure: Probability of detecting at least one fault The P-measure of the k tests T 1, T 2, …, Tk is defined as follows. P(T 1, T 2, …, Tk)=d/k, where k>0, When k , P(T 1, T 2, …, Tk) approaches the probability that a test set T detects at least one fault. Exterimental Study of ART Reliability 15

Aug. 2009 Understanding P-measure When the test cases in each test set T are generated at random independently using the same distribution of the operation profile, the probability that a test set of size w detects at least one fault is 1 -(1 - )w, where is the failure probability of the software. Formally, for random testing, we have that P(T) 1 -(1 - )w, where w = | T |. In other words, the P-measure of random testing is an estimation of the value 1 -(1 - )w, where w is the test size. Exterimental Study of ART Reliability 16

Measurements of fault detecting ability (2) Aug. 2009 E-measure: Average number of failures The E-measure of k tests T 1, T 2, …, Tk is defined as follows. M(T 1, T 2, …, Tk)= where k>0, Exterimental Study of ART Reliability , , i=1, …, k. 17

Aug. 2009 Understanding the E-measure When the test sets Ti are of the same size, say w, and the test cases are selected at random independently using the same distribution as the operation profile, the E-measure approaches w, when k . Here, is the failure probability of the SUT. Formally, for random testing we have that M(T) w, where w = | T |. When test set size is 1, for random testing, P-measure equals to E-measure. In other words, the E-measure of random testing is the estimation of w, where w is the test size. Exterimental Study of ART Reliability 18

Measurements of fault detecting ability (3) Aug. 2009 F-measure: The First failure The F-measure is defined as follows. F(T 1, T 2, …, Tk)= , where k>0, Ti=<a 1, a 2, …, aki>, vi is a natural number that S fails on the test case avi and for all 0<n<vi, S is correct on test case an. It is assumed that the sizes of the test sets Ti, i=1, …, k, are large enough so that vi exists. Exterimental Study of ART Reliability 19

Understanding the F-measure v The F-measure is a random variable of geometric distribution, i. e. the distribution of first success Bernoulli trial, under the conditions: the test cases ai in each test set T are selected at random independently Ø using the same distribution of the operation profile Aug. 2009 Ø other words, when k , F(T 1, T 2, …, Tk) 1/ , where is the probability of failure. F(T) 1/ v In Summary. For random testing: P(T) 1 -(1 - )w M(T) w F(T) 1/ where w= |T| Exterimental Study of ART Reliability 20

Measuring the reliability of test methods S-measure: The variation of fault detecting ability Let: The S-measure is the sample standard deviation of the values m , …, mk of the f-measures on the test sets. T 1, T 2, …, T 1 k be 2 a number of test sets generated according to a testing method the S-measure, the more reliable the testing The smaller method is. measurement of fault detecting ability, Ø f be a given Aug. 2009 Ø The S-measure S(T 1, T 2, …, Tk) of tests T 1, T 2, …, Tk is formally defined as follows. S(T 1, T 2, …, Tk)= where k>1, mi=f(Ti), i=1, …, k, Exterimental Study of ART Reliability , . 21

Andrews, the Briand Labiche (2005): mutants The experiment: goals Aug. 2009 systematically generated by using mutation testing tools can represent the reliabilities real faults. v to evaluate ART methodsvery onwell their of Thus, we use mutants in our experiments rather than fault-detecting abilities with respect to realistic simulation. faults. v to identify the factors that influence the reliability of fault detecting ability. Ø two main candidate factors: § the reliability (or equivalently the failure probability) of the SUT § the regularity of failure domains. This is achieved through the design of experiment with repetition of testing on a number of test sets. Exterimental Study of ART Reliability 22

Experiment process A. Selection of subject programs Ø Ø Aug. 2009 B. Generation of mutants Ø C. Ø For each testing method, five test sets were generated Each test set consists of 1000 test cases. Test the mutants Ø E. Systematically by using the Mu. Java testing tool Generation of test sets Ø D. GCD: the greatest common divisor LCM: the least common multiplier Two dimensional input space on natural numbers. Both F-measures and E-measures are used Analysis of the test results Exterimental Study of ART Reliability 23

Distribution of mutants Aug. 2009 Mutant Types of mutants according to the mutation operators. The trivial mutants fail on every test case. GCD LCM Total AOIS 54 52 106 AOIU 9 6 15 AORB 4 4 8 AORS 0 1 1 LOI 14 14 28 ROR 15 15 30 Total 96 92 187 Equivalent Mutants 28 21 49 Trivial Mutants 27 29 56 Used in experiment 41 52 93 Exterimental Study of ART Reliability 24

Aug. 2009 Test Set Generation Algorithm 1010 Step 1. The input domain is divided into two sub-domains. The original subdomain D 0={1. . 5000} {1. . 10000}, The mirror sub-domain D 1={5001. . 10000} {1. . 10000}. Step 2. Use pseudo-random function of the Java language with different seeds to generate 500 random test cases in the domain D 0. Let T 1 be the set of these test cases, where the elements are ordered in the order that they are generated. Step 3. The mirror test set T 2 is generated by applying the following mirror mapping m: D 0 D 1. 3 10 m(<x, y>)=<x+5000, y>. (*) The test set RT is obtained by merging T 1 with T 2, i. e. adding the elements of T 2 at the end of T 1. Step 4. Let T=T 1 T 2. The test set DMART is obtained by applying the Distance manipulation on T, i. e. DMART=Distance(T). Step 5. Let T 3 be the result of applying the distance manipulation on T 1, i. e. T 3=Distance(T 1). The test set MDART is obtained by applying the Mirror manipulation on T 3 with the mirror mapping m as defined in (*) above, i. e. MDART=Mirror(T 3)=Mirror(Distance(T 1)). Exterimental Study of ART Reliability 25

The characteristics of the mutants Distribution of average E-measures over 5 test sets Aug. 2009 Trivial mutants Exterimental Study of ART Reliability 26

What’s new in the mutants? In the existing research on ART techniques, the subject samples (i. e. the SUT) are always selected with very low failure rates. Ø Mayer, Chen, Huang (2006): the comparison of RT and ART techniques Aug. 2009 § mutants with a failure rate higher than 0. 05 were dismissed. § all the mutants of failure rates at most 0. 05 were treated equally in statistical analysis. Chen et al. (2004): the performances of ART testing techniques were evaluated by simulating the failure rates of 0. 01, 0. 005, 0. 001 and 0. 0005. Ø Chen et al. (2007): most of the simulations are for failure rates less then 25%. There are only three sets of simulation data for failure rates higher than 25%, i. e. , 1/2 and. Open question: How does ART perform on SUT that have higher failure rates. Ø Exterimental Study of ART Reliability Simulation experiments on ART 27

Aug. 2009 Distribution of St. Dev of E-measures over 5 test sets Exterimental Study of ART Reliability 28

Aug. 2009 The relationship between Avg E-measures and their St. Dev Exterimental Study of ART Reliability 29

Main findings : (1) Fault detecting abilities Aug. 2009 Average F-measures over all non-trivial mutants Exterimental Study of ART Reliability 30

How much ART improves RT? Aug. 2009 MDART improves RT by 3. 75%; DMART improves RT by 17. 64%, Exterimental Study of ART Reliability 31

Comparison with existing works v The results we obtained seem inconsistent with existing works obtain by simulations. Aug. 2009 Ø MART and DART were of similar fault detecting abilities in terms of average F-measures. They only differ from each other in the time needed to generate test cases. Ø The scale of improvement is much smaller than that demonstrated in existing work, which could be up to 40% increase in F-measures while we have shown that the average improvement is less than 20%. Exterimental Study of ART Reliability 32

Main findings: (2) Factors affect ability What is the main factors that affect ART’s fault detecting ability? Aug. 2009 three levels of F-measures (>2. 5, 1. 5 ~ 2. 5, <1. 5) the M-measures can be divided into three areas (240~340, 340 ~ 680, >680). Exterimental Study of ART Reliability 33

Aug. 2009 This explains why our results are different from existing works v Our mutants have higher failure rates. v Others’ have lower faiulre rates ART’s fault detecting abilities increase as SUT’s failure rate decreases. Exterimental Study of ART Reliability 34

Aug. 2009 Main findings: (3) ART’s reliability DMART is the most reliable testing method among the three. Exterimental Study of ART Reliability 35

How much ART improves RT on reliability? The improvement that MDART made Aug. 2009 is much smaller DMART made a significant improvement in reliability over RT. Exterimental Study of ART Reliability 36

Main findings: (4) Factors affect reliability Aug. 2009 Is SUT’s failure rate a factor that affects ART reliability? The relationship between the standard deviation of the F-measure on DMART tests and the average failure rates of the mutants. Exterimental Study of ART Reliability 37

Aug. 2009 The variation of DMART tests’ Fmeasure decreases as the average failure rates increases. The correlation coefficients between the average E-measure and the average standard deviations of F-measure of RT, MDART and DMART on all mutants are -0. 735, -0. 645 and -0. 615, respectively. The reliabilities of these testing methods depend on SUT’s failure rate. Exterimental Study of ART Reliability 38

Aug. 2009 Is the ‘regularity’ of failure domain a factor that affects reliability? As the standard deviations of E-measures increases, the standard deviations of F-measures tend to increase. Exterimental Study of ART Reliability 39

Aug. 2009 The average standard deviation is calculated on mutants in various ranges of standard deviations of E-measures The correlation coefficients between the standard deviations of Fmeasures of RT, MDART and DMART and the standard deviations of E-measures for all mutants are 0. 574, 0. 464 and 0. 430, respectively. The reliabilities of testing methods are related to the standard deviation of E-measures, i. e. the regularity of failure domains. Exterimental Study of ART Reliability 40

Conclusion confirmed that ART not only improves the fault detecting ability even on higher failure rate SUT v We discovered that ART significantly improves the reliability of fault detecting ability Aug. 2009 v We Ø the variations on fault detecting abilities are significantly smaller than random testing, where § fault detecting ability is measured by F-measures, § the variation is measured by the standard deviation of F-measures. v We discovered two factors that affect ART’s reliability. Ø the failure rate of the software under test: when the SUT’s failure rate increases, the test method’s reliability increases. Ø the regularity of the failure domain: when the regularity increases, the reliability also increases, where § regularity is measured by the standard deviation of E-measures. Exterimental Study of ART Reliability 41

References Aug. 2009 v Yu Liu and Hong Zhu, An Experimental Evaluation of the Reliability of Adaptive Random Testing Methods, Proc. of The Second IEEE International Conference on Secure System Integration and Reliability Improvement (SSIRI 2008), Yokohama , Japan, July 14 -17, 2008, pp 24 -31. Exterimental Study of ART Reliability 42