Combinatorial Testing Using Covering Arrays Going Beyond Pairwise

Combinatorial Testing Using Covering Arrays - Going Beyond Pairwise Testing Raghu Kacker, NIST Yu Lei, UTA 5/15/06

Outline q Introduction q The IPO Strategy q Fire. Eye: An N-Way Testing Tool q Related Work q Conclusion Combinatorial Testing Using Covering Arrays 2

Why Software Testing? q Modern society is increasingly dependent on the quality of software systems. q A NIST study reports that software failure costs the US economy billions of dollars every year. q Testing (or dynamic analysis) is the most widely used approach to ensuring software quality § Other approaches like static analysis and formal verification are more difficult to apply and do not seem to scale Combinatorial Testing Using Covering Arrays 3

Testing Process The testing process consists of three stages: q Test Generation: Generate test data § For model-based testing, a model (or abstraction) of the system has to be built. q Test Execution: Test setup and the actual test runs q Test Evaluation: Check if the output is in line with expectations Combinatorial Testing Using Covering Arrays 4

All About Trade-Off q Testing is labor intensive and can be very costly § estimated to often consume more than 50% of the development cost q Exhaustive testing is often impractical due to resource constraints q From a certain perspective, testing is basically about making a good trade-off between test effort and quality assurance. Combinatorial Testing Using Covering Arrays 5

Combinatorial Testing q Generates tests from an input parameter model by combining the values of the parameters. § § § Requires lightweight specification and no knowledge about the implementation structure Can be virtually applied to any software and at different levels of abstraction Can be implemented as a push-button feature q Usually performed to achieve t-way coverage, i. e. guarantees to cover every t-way interaction § § Motivation: Not every parameter contributes to every fault Can dramatically reduce the number of tests while still preserving important fault detection capabilities. Combinatorial Testing Using Covering Arrays 6

The ASTUCA Project q Develop new methods and tools for efficient t-way testing for up to 6 -way testing § § Create new algorithms for efficient test set construction Provide adequate support for parameter relations and constraints Explore integration with other software testing tools Conduct empirical evaluation of t-way testing in an industrial setting q A collaborative effort among the following institutions: § § § The US National Institute of Standards and Technology George Mason University The University of Texas at Arlington Combinatorial Testing Using Covering Arrays 7

State-of-the-Art q Existing work has mainly focused on pairwise testing q Many failures are caused by the interaction involving more than two parameters § For certain software, pairwise testing discovers a relatively low percentage of faults • e. g. , For the RAX in NASA Deep Space 1 mission, pairwise testing only discovers 54 percent of interface faults, and 47 percent of engine faults. q Increased coverage leads to a higher level of assurance § Many applications, e. g. , security protocols, have strict requirements on test coverage Combinatorial Testing Using Covering Arrays 8

Technical Challenges q The computational complexity for t-way testing grows rapidly as the value of t increases § New algorithms must strike a balance between the time and space requirements and the optimality of the resulting test sets q The number of tests also grows rapidly as the value of t increases § § Impractical to manually execute and inspect the results of a large number of test runs Test generation, test execution, and test evaluation must integrate together to enable test automation Combinatorial Testing Using Covering Arrays 9

Terminology q N-Way Test set -> N-Way Covering array q Tests -> Rows q Parameters -> Factors or Columns q Values -> Levels Combinatorial Testing Using Covering Arrays 10

Outline q Introduction q The IPO Strategy q Fire. Eye: An N-Way Testing Tool q Related Work q Conclusion Combinatorial Testing Using Covering Arrays 11

The Framework (1) q Builds a t-way test set in an incremental manner § A t-way test set is first constructed for the first t parameters, § Then, the test set is extended to generate a t-way test set for the first t + 1 parameters § The test set is repeatedly extended for each additional parameter. q Two steps involved in each extension for a new parameter: § § Horizontal growth: extends each existing test by adding one value of the new parameter Vertical growth: adds new tests, if necessary Combinatorial Testing Using Covering Arrays 12

The Framework (2) Strategy In-Parameter-Order begin /* for the first t parameters p 1, p 2 , …, pt*/ T : = {(v 1, v 2, …, vt) | v 1, v 2, …, vt are values of p 1, p 2, …, Pk, respectively} if n = t then stop; /* for the remaining parameters */ for parameter pi, i = t + 1, …, n do begin /* horizontal growth */ for each test (v 1, v 2, …, vi-1) in T do replace it with (v 1, v 2, …, vi-1, vi), where vi is a value of pi /* vertical growth */ while T does not cover all the interactions between p i and each of p 1, p 2, …, pi-1 do add a new test for p 1, p 2, …, pi to T; end Combinatorial Testing Using Covering Arrays 13

Example (1) Consider a system with the following parameters and values: q parameter A has values A 1 and A 2 q parameter B has values B 1 and B 2, and q parameter C has values C 1, C 2, and C 3 Combinatorial Testing Using Covering Arrays 14

Example (2) A A 1 A 1 A 2 A 2 B B 1 B 2 C C 1 C 2 C 3 C 1 Horizontal Growth Combinatorial Testing Using Covering Arrays A A 1 A 2 A 2 A 1 B B 1 B 2 C C 1 C 2 C 3 Vertical Growth 15

Comparison to AETG (1) q A commercial tool developed by Telcordia, and protected by a US patent q Starts with an empty set and adds one (complete) test at a time q Each test is locally optimized to cover the most number of missing pairs: § § § Generate a random order of the parameters Use a greedy algorithm to construct a test that covers the most uncovered pairs Repeat the above two steps for a given number of times (suggested 50), and select the best one Combinatorial Testing Using Covering Arrays 16

Comparison to AETG (2) A B C A 1 B 1 C 1 Adds the 1 st test A B C A 1 B 1 C 1 A 1 B 2 C 2 Adds the 2 nd test Combinatorial Testing Using Covering Arrays A A 1 A 2 A 2 A 1 B B 1 B 2 C C 1 C 2 C 3 Adds the last test 17

IPO vs AETG q IPO is deterministic, whereas AETG is inherently non-deterministic q IPO has a lower order of complexity, both in terms of time and space, than AETG § § IPO constructs a test set one parameter at a time and in a more incremental nature. The results generated by IPO are still competitive to those generated by AETG. q IPO is more flexible than AETG § IPO can take a test set for a subsystem (i. e. , for a subset of parameters) and then extend it to a complete set for the entire system Combinatorial Testing Using Covering Arrays 18

Outline q Introduction q The IPO Strategy q Fire. Eye: A Prototype Tool q Related Work q Conclusion Combinatorial Testing Using Covering Arrays 19

Major Features q Uses Java as the programming language § Relatively easier to program, leading to reduced development time and ease of maintenance § Supports the concept of “write-once-run-everywhere” q Data structures are carefully designed to optimize the runtime performance § A hierarchical structure is used to manage the possible interactions q Allows an incomplete test set to be extended to a complete one § Add new tests or parameters, if necessary, to achieve tway coverage Combinatorial Testing Using Covering Arrays 20

Initial Results (1) [System] Name: Test Configuration for TCAS [Parameter] -- only compare with MINSEP and MAXALTDIFF Cur_Vertical_Sep : 299, 300, 601 High_Confidence : TRUE, FALSE Two_of_Three_Reports_Valid : TRUE, FALSE -- Low and High, only compare with Other_Tracked_Alt Own_Tracked_Alt : 1, 2 Other_Tracked_Alt : 1, 2 -- only compare with OLEV Own_Tracked_Alt_Rate : 600, 601 Alt_Layer_Value : 0, 1, 2, 3 -- compare with each other (also see NOZCROSS) and with ALIM Up_Separation : 0, 399, 400, 499, 500, 639, 640, 739, 740, 840 Down_Separation : 0, 399, 400, 499, 500, 639, 640, 739, 740, 840 Other_RAC : NO_INTENT, DO_NOT_CLIMB, DO_NOT_DESCEND Other_Capability : TCAS_TA, OTHER Climb_Inhibit : TRUE, FALSE Combinatorial Testing Using Covering Arrays 21

Initial Results (2) All the experiments are performed on a desktop with 1. 2 GHZ CPU and 1 GB memory. Combinatorial Testing Using Covering Arrays 22

Outline q Introduction q The IPO Strategy q Fire. Eye: An N-Way Testing Tool q Related Work q Conclusion Combinatorial Testing Using Covering Arrays 23

Classification q Search-Based methods that are mainly developed by computer scientists § AETG (from Telcordia), TCG (from JPL/NASA), DDA (from ASU), Pair. Test q Algebraic methods that are mainly developed by mathematicians § § Orthogonal Arrays Recursive Construction Combinatorial Testing Using Covering Arrays 24

Orthogonal Arrays q Orthogonal arrays can be constructed very fast and are always optimal § Any extra test will cause a pair to be covered for more than once q However, there are several limitations: § Orthogonal arrays do not always exist § Every parameter must have the same number v of values § Existing methods often require v be a prime power. § Every t-way interaction must be covered at the same number of times Combinatorial Testing Using Covering Arrays 25

Recursive Construction q Covering arrays are a more general structure, which requires every t-way interaction be covered at least once q Constructing a covering array from one or more covering arrays with smaller parameter sets q Recursive construction can be fast, but it also has restrictions on the number of parameters and the domain sizes Combinatorial Testing Using Covering Arrays 26

Search-Based vs Algebraic Methods q Search-based methods: § Advantages: no restrictions on the input model, and very flexible, e. g. , relatively easier to support parameter relations and constraints § Disadvantages: explicit search takes time, the resulting test sets are not optimal q Algebraic methods: § Advantages: very fast, and often produces optimal results § Disadvantages: limited applicability, difficult to support parameter relations and constraints q The advantages and disadvantages of the two types of methods seem to complement with each other Combinatorial Testing Using Covering Arrays 27

Outline q Introduction q The IPO Strategy q Fire. Eye: An N-Way Testing Tool q Related Work q Conclusion Combinatorial Testing Using Covering Arrays 28

Conclusion q Combinatorial testing is a well-defined problem and has been used widely in practice. q The IPO strategy has a lower order of complexity than AETG, and still produces competitive results. q Algebraic methods, if applicable, are fast and can be optimal, whereas search-based algorithm are very flexible. q Going beyond 2 -way testing presents challenges and opportunities to the area of combinatorial testing. Combinatorial Testing Using Covering Arrays 29

References 1. Boroday S. Y. and Grunskii I. S. , “Recursive generation of locally complete tests, ” Cybernetics and Systems Analysis 28 (1992), 20 -25. 2. K. A. , Bush, “Orthogonal arrays of index unity, ” Annals of Mathematical Statistics, 23 (1952), 426 -434. 3. D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton, “The AETG System: An Approach to Testing Based on Combinatorial Design, ” IEEE Transactions on Software Engineering, 1997, Vol. 23, No. 7. 4. M. B. Cohen, C. J. Colbourn, P. B. Gibbons and W. B. Mugridge, “Constructing test suites for interaction testing, ” In Proc. of the Intl. Conf. on Software Engineering, (ICSE 2003), 2003, pp. 38 -48, Portland. 5. R. Kuhn, D. Wallace, A. Gallo, “Software Fault Interactions and Implications for Software Testing, ” IEEE Transactions on Software Engineering, June 2004, Vol. 30, No. 6. Alan Hartman, Leonid Raskin, “Problems and algorithms for covering arrays, ” Discrete Mathematics 284(1 -3): 149 -156 (2004) 7. Y. Lei and K. C. Tai , “In-parameter-order: a test generation strategy for pairwise testing, ” Proceedings Third IEEE Intl. High-Assurance Systems Engineering Symosium. , 1998, pp. 254 -261. 8. K. C. Tai and Y. Lei, “A Test Generation Strategy for Pairwise Testing, ” IEEE Transactions on Software Engineering, 2002, Vol. 28, No. 1. Combinatorial Testing Using Covering Arrays 30