Testcover com The Bug Stops Here Improving Test
Testcover. com The Bug Stops Here Improving Test Design with a Novel Test Case Generator Technical Presentation Salon 6 Wednesday May 18, 1: 45 – 2: 45 Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com How to Spin a Web of Effective Test Cases • • • Test case generator makes “all pairs” testing simpler, quicker, more practical New technology improves test case coverage and fault discovery Service continually integrates research advances to reduce number of test cases Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Testing Breakthrough Needed • • • Increasing dependency on software, networks and distributed applications Increasing expectation and need for quality Facts of life: • More features and complexity • Shorter development cycles • More reuse of components Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com This Presentation • Describes approaches to software test case generation • Test configurations • System-level functional tests – integration tests, functional verification tests, regression tests etc. • Does not address • Unit testing (e. g. code coverage) • Stability and load testing Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com This Presentation • • Explains how the Testcover. com Covering Test Case Generator Service improves test case coverage and fault discovery Shows how to use the service with examples Tells how to receive 1 month service free Is available on the web at Testcover. com Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Dedicated to Excellence in Test Design • • Founded in 2003 to aid testers in the selection of efficient sets of test cases Covering Test Case Generator Service On the web at Testcover. com Technology based on: • advances in covering array construction • practical software engineering experience Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Configuration Example • Test three new applications on a networked computer • Test factors and their values: Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Configuration Example • A simple approach: • One test configuration for each factor value • Select a most common or default value for each factor • Vary each factor over all its values while the other factors are set to their default values Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Configuration Example • A simple approach: • How many test configurations? 12 Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Configuration Example • A simple approach: • Small number of test configurations • All individual factor values are covered • Many pairs of factor values are not: • Application 2 with Cable Connection • Firefox Browser with ME Operating System • Application 3 with Medium Resolution • More… Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Configuration Example • A better approach: • Include all pairs of factor values • Test all pair-wise interactions • How many pairs of factor values? 101 • How many test configurations? 16 Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Configuration Example How many pairs of factor values? Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Configuration Example How many pairs of factor values? Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Configuration Example How many pairs of factor values? Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Configuration Example How many pairs of factor values? Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Configuration Example How many pairs of factor values? Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Configuration Example How many pairs of factor values? Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Configuration Example How many pairs of factor values? Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Configuration Example How many pairs of factor values? Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Configuration Example How many pairs of factor values? Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Configuration Example How many pairs of factor values? Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Configuration Example • 16 configurations cover all 101 pairs Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Combo Countdown • • • Coverage metric Number of untested pairs after each test case Stops at 0 or the number of untested pairs when there are constraints Illustrates diminishing returns Supports risk assessment Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Test Definitions • • Test factor – any variable whose values are to be controlled during testing Test factor value – specific value taken by one of the test factors Test case – set of test factor values with one allowed value for each of the test factors Combination – association of some number of factor values (pairs, triples, etc. ) Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Array Definitions • • Size – number of test cases, number of rows in array Degree – number of test factors, number of columns in array Order – number of values for a test factor Strength – size of combinations (2: pairs, 3: triples, etc. ) Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Mixed arrays • • Mixed array has factors with different orders Example: 52 43 32 21 • • 2 3 2 1 factors with 5 values factors with 4 values factors with 3 values factor with 2 values • Degree = 2 + 3 + 2 + 1 = 8 • There is a 52 43 32 21 array with strength 2 and size 25 Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Early Approaches • • • Orthogonal arrays used in agricultural and medical experiments Dr. Genichi Taguchi used orthogonal arrays to select test parameters Statistical coverage where exhaustive testing is not possible Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Real-world Constraints • • • What if the applications in the configuration example are ported to Linux and Macintosh OSX? Internet Explorer is supported on Windows only There are Browser-Operating System pairs that must be excluded Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Configuration Example • The wrong answer: Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Constrained Array Test System • • • Motivated by testing Local Area Network product configurations Developed in 1990 to handle constraints Employed greedy search algorithm Used internally at AT&T as best practice Described in STAR proceedings May 1994 Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Current Challenges 1. 2. 3. 4. Find small array quickly Adapt to system constraints simply Support multiple system states Allow easy use Testcover. com addresses these challenges Let’s see how… Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Sample Arrays Size Array 6 210 23 48 61 710 8 230 28 424 91 730 11 35 33 58 78 811 15 320 45 530 120 830 19 330 46 69 81 910 16 45 76 627 153 930 Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Sample Mixed Arrays Size Degree Array 15 30 36 224 12 10 41 3 3 2 6 22 25 45 220 20 17 51 44 212 30 25 45 320 25 8 52 4 3 3 3 27 23 53 320 30 30 61 55 224 39 23 63 320 36 15 62 51 312 49 23 73 420 42 15 71 62 312 Copyright © 2005 Testcover. com, LLC. All rights reserved.
The results given here are subject to improvement without notice! Testcover. com Challenge 1: Fewer Test Cases • • • Small arrays result from multiple techniques Testcover. com continually integrates research advances to reduce the numbers of test cases New technology provides quicker results Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Challenge 2: Handle Constraints • • Direct Product Block notation enables simple descriptions of complex constraints and dependencies Let’s see how DPB notation works with the Configuration Example Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Configuration Example • No constraints – one block Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Direct Product Block Notation • • • Block represents all possible test cases for selected values of the test factors Multiple blocks represent allowed test cases – as a set of products Simpler than enumerating all disallowed combinations Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Configuration Example • Browser/OS constraints – two blocks Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Challenge 3: Multiple States • Calendar example illustrates: • Direct Product Block notation • Constraints – different month lengths • Multiple states using different partitions of test cases – • Normal (good dates) • Error handling (too long months) Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Calendar Example Request Part 1 Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Calendar Example Request Part 2 Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Calendar Example Results Part 1 Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Calendar Example Results Part 2 Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Calendar Example Results Part 3 Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Challenge 4: Ease of Use • Direct Product Block notation • Simple to learn • Flexible and efficient • Internet-based service • Easy access without special installation • Independent of development environment • Enhancements without upgrade work • Practical for everyday use Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Challenge 4: Ease of Use • Advanced Web Search example illustrates: • • Practical for real HTML form Direct Product Block notation Constraints – language button use Small number of test cases Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Advanced Web Search Example • See HTML form Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Advanced Web Search Example • Request part 1 Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Advanced Web Search Example • Request part 2 Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Advanced Web Search Example • Request part 3 Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Advanced Web Search Example • • 81 72 61 45 34 28 Array Degree 21 2715 pairs, 6 not covered (constraints) Size 63 (compared to 8 x 7 = 56) Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com How to Spin a Web of Effective Test Cases • • • New technology improves test case coverage and fault discovery Service continually integrates research advances to reduce number of test cases Exclusive Direct Product Block notation simplifies use Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Exhibit at Booth 8 • • • See these and other examples run Receive your brochure and gift Register for 1 free month of service Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com STAREAST 2005 First Month Free 4 Month Subscription for the Price of 3 • • For STAREAST 2005 Attendees Only Register at Booth 8 May 18 – May 19 Try Testcover. com Service Free for 1 Month Your Choice: • Pay invoice for 4 -Month Subscription • Cancel On-line • 1 Month Free Service either way – $100 Value Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Back-up Slides Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Test Case Objectives • • Run enough of the “right” test cases to exercise system interactions completely Avoid wasting time and money on unnecessary test cases Insure system behavior is thoroughly verified Allow for complex and changing system requirements Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Orthogonal Arrays & Covering Arrays • Every pair of columns in an orthogonal array contains all pairs of values an equal number of times • • • Every pair of columns in a covering array contains all pairs of values at least once Orthogonal arrays are covering arrays Both types can be mixed (different orders) Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Orthogonal Arrays & Covering Arrays • Orthogonal arrays are balanced – suggest which factor values give best measured results • Tuning complex systems • Medical & agricultural trials • Covering arrays are not required to be balanced – can be smaller for pass/fail type tests Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Covering Arrays vs. Random Values • • Goal: Validate operation quickly – integration, function, regression – Arrays provide coverage with few test cases Goal: Validate continuous operation – stability, load, stress – Random values can provide many different test cases Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Higher Strength Arrays • • • Strength t arrays are defined similarly – Every set of t columns in an array contains all t-tuples of values (3 -tuples, 4 -tuples, etc. ) Size (number of test cases) increases with strength – Minimum size is product of the t largest orders Example: 52 43 32 21 arrays Strength: Minimum Size: 2 3 4 25 100 400 Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Higher Strength Arrays • • Higher strength arrays can cover more combinations of factor values, with a cost of more test cases Generally finding higher strength arrays requires more processing See Sherwood, Martirosyan, and Colbourn, Sections 4 & 5. Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com RDF/XML Results • • • The Resource Description Framework (RDF) is a generalpurpose language endorsed by the World Wide Web Consortium (W 3 C) for representing information on the web. RDF is based on the idea of identifying things using Uniform Resource Identifiers (URIs) and describing resources in terms of simple properties. RDF represents simple statements about resources as a graph of nodes and arcs representing the resources, and their properties and values. RDF also provides an XML-based syntax (called RDF/XML) for recording and exchanging these graphs. In addition to HTML results, the Testcover. com test case generator provides RDF/XML results for use by other applications. Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Sources & Further Reading Companies Seek to Hold Software Makers Liable for Flaws, David Bank, The Wall Street Journal, page B 1, February 24, 2005. To Darned Big to Test, Keith Stobie, ACM Queue, vol. 3, no. 1, pages 30 -37, February 2005. Products of Mixed Covering Arrays of Strength Two, C. J. Colbourn, S. S. Martirosyan, G. L. Mullen, D. Shasha, G. B. Sherwood, and J. L. Yucas, preprint, 2004. Covering Arrays of Higher Strength From Permutation Vectors, G. B. Sherwood, S. S. Martirosyan, and C. J. Colbourn, preprint, 2004. On the Construction of Orthogonal Arrays and Covering Arrays Using Permutation Groups, George Sherwood, http: //home. att. net/~gsherwood/cover. htm March 2002 - June 2004. Copyright © 2005 Testcover. com, LLC. All rights reserved.
Testcover. com Sources & Further Reading Effective Testing of Factor Combinations, George Sherwood, Third International Conference on Software Testing, Analysis & Review , May 8 -12, 1994, Washington, DC. Orthogonal Arrays Theory and Applications, A. S. Hedayat, N. J. A. Sloane, and John Stufken, 1999 Springer-Verlag. Quality Engineering Using Robust Design, M. S. Phadke, 1989, Prentice Hall. M. J. Suárez-Cabal and J. Tuya. Using an SQL Coverage Measurement for Testing Database Applications. ACM Software Engineering Notes. 29(6): 253 -262, 2004. Copyright © 2005 Testcover. com, LLC. All rights reserved.
- Slides: 64