Testcover com Pairwise Testing Comes of Age George

Testcover. com Pairwise Testing Comes of Age George B. Sherwood, Ph. D. Thursday May 8, 1: 30 p. m. T 14 Pairwise Testing Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Pairwise Testing Comes of Age Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Évariste Galois • • French mathematician, studied abstract algebra & group theory Radical Republican during the reign of Louis Philippe Short life with tragic ending: 1829 • 1831 • Father committed suicide • Failed entry to École Polytechnique • 1830 • Expelled from École Normale for criticizing director • Imprisoned for wearing illegal uniform • 1832 • Rejected by Stéphanie. Felice du Motel • Shot in duel at age 20 Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com 21 st Century Testing • • • Increasing dependency on software, networks and distributed applications Increasing expectation and need for quality Facts of life: • More features and complexity • Shorter development cycles • More reuse of components Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Why Pairwise Testing? • Statistical coverage to find faults Improved quality • Small number of test cases Reduced cost Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Pairwise Testing Example • • Test 3 new applications on a networked computer 4 test configuration factors with 3 values 34=81 configurations in total – too many 3 configurations cover all values individually – but only a few interactions Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Pairwise Testing • • Includes all pairs of interactions between test factors Provides a small set of test cases – a practical alternative to testing all combinations Started almost 20 years ago Increasingly popular way to improve quality and reduce cost Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Galois Field Example • Modulo 3 arithmetic – GF(3) Usual + or x, divide by 3 & take remainder • Use GF(3) tables to make orthogonal array • Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Orthogonal Array OA(9; 33) • Replace each element in the x table with the corresponding column from the + table Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Orthogonal Array OA(9; 34) • Add one column which pairs a different value with every 3 rows Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Networked Computer Example • • 4 test configuration factors with 3 values Replace numbers in OA(9; 34) template with test factor values Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Networked Computer Example • Only 9 configurations are needed to cover all 54 pairs of values using OA(9; 34) Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Design of Experiments • • • Ronald A. Fisher analyzed data from crop yield experiments Developed new statistical methods to study experimental data Pioneered the mathematical principles of experimental design Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Orthogonal Arrays • • • Calyampudi Radhakrishna Rao introduced Orthogonal Arrays Orthogonal Array designs are used in agricultural & manufacturing experiments, clinical trials, market research… Results can show effects of factors and how they interact Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com OAs for Manufacturing • • • Genichi Taguchi adapted Fisher’s experimental design to improve quality of manufactured goods Used OAs to select test parameters in creative ways Worldwide influence in improving manufactured quality Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com OAs for Software Test • • • Keizo Tatsumi, Robert Mandl & others applied these ideas to software testing Taguchi visited Bell Labs and collaborated with Madhav Phadke Orthogonal Array Test System (OATS) offered statistical coverage where exhaustive testing was not possible Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Star. LAN and Star. GROUP • Client installation test • 101 combinations of PCs and OS versions • 15 network cards • • • Tried to use OATS to reduce number of test configurations OATS could not handle our complex configuration constraints correctly Generated impossible tests, as in previous example Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Networked Computer Example • Two of the 9 configurations from OA(9; 34) are invalid Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com What to do? • • • Skip the bad cases? Then for each case there is no coverage for 5 good pairs, along with the invalid one Replace values ‘Linux’, ‘OS X’ or ‘Int Explorer’ to avoid bad pairs? Still miss good pairs Solution: Computer search for small number of cases to cover allowed pairs. Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Networked Computer Example • 11 configurations cover 52 allowed pairs Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Constrained Array Test System • • • Search among allowed test cases to handle constraints Find next “best” test case to minimize uncovered pairs Secondary result: fewer test cases than OATS Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Types of Arrays Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Early Tools Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Pairwise Selection of Input Data • Identify equivalence classes • Analyze functional requirements, use cases • Partition factor value combinations for equivalent processing and expected results • Generate test cases for different classes separately • All pairs from each class go to same resulting state • Example: Don’t mix error combinations with normal combinations Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Uncle Katzendogg’s Guaranteed Long-term Weather Forecasts Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Weather Forecast Example • • 4 test factors: Month, Day, Year, Zip code 3 sets of test cases leading to different states: 1. Valid dates with valid zip codes Forecast returned 2. Invalid dates with valid zip codes Date error returned 3. Valid dates with invalid zip codes Zip code error returned Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Weather Forecast Valid Input Factor Values Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Weather Forecast Valid Input Test Cases Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Weather Forecast Invalid Date Factor Values Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Weather Forecast Invalid Date Test Cases Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Weather Forecast Invalid Zip Code Factor values Test cases Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Pairwise Testing Today • • Test demands continue to grow Pairwise testing is a popular response for effective coverage with a limited number of test cases Improved training is needed, e. g. to handle constraints, equivalence classes Careless use has led to controversy Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Pairwise. org Tools List – part 1 Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Pairwise. org Tools List – part 2 Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Bach & Schroeder • “Pairwise Testing: A Best Practice that Isn’t” • Useful criticism of careless pairwise testing • MS Word options example – 31212 Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Pairwise Testing Efficiency • • • Pairwise generators are more efficient than random test case selection Significant improvements in pairwise efficiency are possible Practitioners need pairwise testing to meet modern testing challenges Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com How do Searches Work? • • Go to the space of all possible test cases Look for small number of test cases (short path) to minimize the uncovered pairs Uncovered pairs Test cases Factor values ? Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Search Variations • • Several algorithms for different tradeoffs Test practitioner goals: • • Few test cases Quick responses Practicality – mixed values, constraints Researcher goals: • • • Fewer test cases More factors New methods, better solutions Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Finding the Minimal Set of Test Cases is Hard • • • ? Typically searches are CPU bound Improvement diminishes with processing time Mixed covering array constructions can complement searches using less processing Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Change Mixed CA Values Without Searching • • • Example: Change OA(9; 34) to CA(12; 413122) Reduce the number of values in two of the factors Increase the number of values in one factor Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com OA(9; 34) to CA(9; 3222) • Replace all the 2 s in any 2 factors with unassigned (*) values Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com CA(9; 3222) to CA(12; 413122) • • Add 3 rows with 3 s to one of the factors with 3 values For each of the other factors, pair its values with the 3 s in the new rows Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com More CA Factors from Products and Searches • Product sequence: • OA(9; 34), CA(15; 315), CA(21; 357), … • Nurmela’s tabu search: • CA(15; 320) • Products using tabu search result: • CA(15; 320), CA(21; 374), CA(27; 3282), … • Combined methods give better results Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Optimal Mixed CAs & More Factors • • • Add n test cases to OA(25; 56) to get optimal array Each factor of 5 values can be replaced by the indicated factors without additional test cases Example: Replace 3 factors in CA(35; 7155) to get CA(35; 715248360) Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Mixed Covering Arrays • Compared with OA templates • More choices for numbers of values • Fewer test cases typically • Compared with greedy search • Faster responses (less processing) • Fewer test cases when CAs are optimal • Search needed for constraints Copyright © 2008 Testcover. com, LLC. All rights reserved.

Testcover. com Future Pairwise Trends • Platforms: • Faster, distributed processing • Tool Software: • Smarter searches, less processing • Fewer test cases • Improved test efficiency • Faster responses • What-if test designs, test changes • Use with agile, incremental cycles Copyright © 2008 Testcover. com, LLC. All rights reserved.