Exercise Using The Pairwise Tool PICT Philippe CHARMAN

Preliminary • Suppose a function f takes five Boolean variables as arguments: x 1,

Overview PICT • PICT is a pairwise testing tool developed by Microsoft • PICT

Installing PICT • Install PICT from pict 33. msi • Update the environment variable

Model files • A model is composed of 3 sections: parameter definitions [sub-model definitions]

Simple Model • Syntax of the parameter definitions <Param. Name> : <Value 1>, <Value

Generating Test Cases • Given this model C: >more mymodel. txt X: 1, 2

Configuration test cases • Suppose we want to test on different platforms: C: >more

Conditional constraints • A term [parameter] relation value is an atomic part of a

Conditional constraints • Parameters can be compared to other parameters, as in this example:

Unconditional constraints (Invariants) • An invariant declares an always-valid limitation of a domain: #

Parameter Types • PICT uses the concept of parameter types. There are two types

Exercise • An application is available in 32 -bit and 64 -bit and can

Writing the model • Write a model to generate pairwise test cases • Trivial

Configuration test cases • In the solution proposed: OS Win C++ Java x 64

Sub Models • Syntax of the sub-model definitions: {<Param. Name 1>, <Param. Name 2>,

How the generation would look for the previous model $ | order = 2

Sub Models • You can define as many sub-models as you want; any parameter

Sub Models C: > more conf. txt OS: x 86, x 64 Win: Win

Sub Models C: > more conf 2. txt x: 0, 1 y: 2, 3

Sub Models C: >pict conf 2. txt x y z t 1 2 5

Defining a sub-model • In the model, express the pair (Application, Architecture, OS) as

Trying to reduce the test case • Still too many test cases • Among

Randomized generation • C: more try. txt x: 0, 1 y: 0, 1 z:

Randomized generation C: >pict try 1. txt /r: 1 Used seed: 1 x y

Generating Test Cases • N-wise test cases can be generated with /o: N C:

Further reading • More information on PICT: http: //msdn. microsoft. com/en-us/library/cc 150619. aspx •

Slides: 28

Download presentation

Exercise: Using The Pairwise Tool PICT Philippe CHARMAN charman@fr. ibm. com http: //users. polytech. unice. fr/~charman/ Last update: 10 April 2015

Preliminary • Suppose a function f takes five Boolean variables as arguments: x 1, x 2, x 3, x 4 and x 5. • Boolean values are expressed as 0 and 1 for simplicity reason • Design the smallest test suite that meets the pairwise property x 1 x 2 x 3 x 4 x 5 0 0 0 ? ? ?

Overview PICT • PICT is a pairwise testing tool developed by Microsoft • PICT is the acronym of Pairwise Independent Combinatorial Testing • Used to design test cases and test configurations for software systems • PICT generates a compact set of parameter value choices that represent the test cases you should use to get comprehensive combinatorial coverage of your parameters.

Installing PICT • Install PICT from pict 33. msi • Update the environment variable PATH • Help in PICTHelp. htm

Model files • A model is composed of 3 sections: parameter definitions [sub-model definitions] [constraint definitions] • Model sections should always be specified in the order shown above and cannot overlap • The sub-model and the constraint sections are optional • Sections do not require any special separators between them • Empty lines can appear anywhere • Comments can be included by prefixing lines with the “#”character.

Simple Model • Syntax of the parameter definitions <Param. Name> : <Value 1>, <Value 2>, <Value 3>, . . . • Sample model for testing volume create/delete functions Type: Primary, Logical, Single, Span Size: 10, 100, 500, 1000, 5000, 10000 Format method: quick, slow File system: FAT, FAT 32, NTFS Cluster size: 512, 1024, 2048, 4096, 8192, 16384 Compression: on, off

Generating Test Cases • Given this model C: >more mymodel. txt X: 1, 2 Y: A, B, C Z: 1, 2, by default pairwise test cases are generated: C: >pict mymodel. txt X Y Z 2 B 2 1 A 1 2 C 1 2 A 2 1 B 1 1 C 2

Configuration test cases • Suppose we want to test on different platforms: C: >more conf. txt OS: x 86, x 64 Win: Win 7, Win 8 C++: msvc 10, msvc 11 Java: JDK 6, JDK 7 by default: C: >pict conf. txt OS Win C++ Java x 64 Win 7 msvc 10 JDK 6 x 64 Win 8 msvc 11 JDK 7 x 86 Win 8 msvc 10 JDK 7 x 86 Win 7 msvc 11 JDK 7 x 86 Win 8 msvc 11 JDK 6

Conditional constraints • A term [parameter] relation value is an atomic part of a predicate. The following relations can be used: =, <>, >, >=, <, <=, and LIKE is a wildcard-matching operator (* - any character, ? – one character). [Size] < 10000 [Compression] = "OFF" [File system] like "FAT*" • Operator IN allows specifying a set of values that satisfy the relation explicitly: IF [Cluster size] in {512, 1024, 2048} THEN [Compression] = "Off"; IF [File system] in {"FAT", "FAT 32"} THEN [Compression] = "Off";

Conditional constraints • Parameters can be compared to other parameters, as in this example: # Machine 1 OS_1: Win 2000, Win. XP SKU_1: Professional, Server, Datacenter, Win. Powered LANG_1: EN, DE # Machine 2 OS_2: Win 2000, Win. XP SKU_2: Professional, Server, Datecenter LANG_2: EN, DE IF [LANG_1] = [LANG_2] THEN [OS_1] <> [OS_2] AND [SKU_1] <> [SKU_2];

Unconditional constraints (Invariants) • An invariant declares an always-valid limitation of a domain: # At least one parameter must be different to be a meaningful test case (use the OR operator) [OS_1] <> [OS_2] or [SKU_1] <> [SKU_2] or [LANG_1] <> [LANG_2]; # All parameters must different (use the AND operator) [OS_1] <> [OS_2] and [SKU_1] <> [SKU_2] and [LANG_1] <> [LANG_2];

Parameter Types • PICT uses the concept of parameter types. There are two types of parameters: string and numeric. A parameter is considered numeric only when all its values are numeric. If a value has multiple names, only the first one counts. Types are only important when evaluating constraints. You can only compare a numeric parameter to a number, and a string parameter to another string. For example: Size: 1, 2, 3, 4, 5 Value: a, b, c, d IF [Size] > 3 THEN [Value] > "b"; • String comparison is lexicographical and case-insensitive by default. You can make comparisons case-sensitive if you specify the /c option. Numeric values are compared as numbers.

Exercise • An application is available in 32 -bit and 64 -bit and can be used on the following platforms: – – Windows 7 – x 86 and x 64 Windows 8 – x 86 and x 64 AIX 6. 1 - Power 7 32 -bit and 64 -bit AIX 7. 1 – Power 7 32 -bit and 64 -bit • The following locales are supported: EN, CN, JP, DE, FR • The following databases are supported: Oracle 9 i, Oracle 10 g, Oracle 11 g, DB 2 9. 1, DB 2 9. 7

Writing the model • Write a model to generate pairwise test cases • Trivial constraints: – The 32 -bit application should be tested on the 32 -bit and 64 -bit architectures. – The 64 -bit application cannot be tested on a 32 -bit architecture • About 27 test cases should be generated

Configuration test cases • In the solution proposed: OS Win C++ Java x 64 Win 7 msvc 10 JDK 6 x 64 Win 8 msvc 11 JDK 7 x 86 Win 8 msvc 10 JDK 7 x 86 Win 7 msvc 11 JDK 7 x 86 Win 8 msvc 11 JDK 6 • No tests are done on x 64/Win 7 to test msvc 11 or x 64/Win 8 to test msvc 10 • Testing the pair (OS, Win) is more important than testing the pair (C++, Java)

Sub Models • Syntax of the sub-model definitions: {<Param. Name 1>, <Param. Name 2>, . . . } @ <Order> • Example: PLATFORM: x 86, ia 64, amd 64 CPUS: Single, Dual, Quad RAM: 128 MB, 1 GB, 4 GB, 64 GB HDD: SCSI, IDE OS: NT 4, Win 2 K, Win. XP, Win 2 K 3 IE: 4. 0, 5. 5, 6. 0 APP: SQLServer, Exchange, Office { PLATFORM, CPUS, RAM, HDD } @ 3 { OS, IE } @ 2

How the generation would look for the previous model $ | order = 2 (defined by /o) | +---------------+---------------+ | order = 3 | order = 2 | { PLATFORM, CPUS, RAM, HDD } { OS, IE } APP

Sub Models • You can define as many sub-models as you want; any parameter can belong to any number of sub-models. Model hierarchy can be just one level deep • The combinatory order of a sub-model cannot exceed the number of its parameters. In the example above, an order of the first sub-model can be any value between one and four • If you do not specify the order for a sub-model, PICT uses the default order or the order specified by /o option

Sub Models C: > more conf. txt OS: x 86, x 64 Win: Win 7, Win 8 C++: msvc 10, msvc 11 Java: JDK 6, JDK 7 {OS, Win} @ 2 C: >pict conf. txt OS Win C++ Java x 86 Win 8 msvc 10 JDK 7 x 86 Win 7 msvc 11 JDK 7 x 86 Win 7 msvc 10 JDK 6 x 64 Win 7 msvc 11 JDK 7 x 86 Win 8 msvc 11 JDK 6 x 64 Win 7 msvc 10 JDK 6 x 64 Win 8 msvc 11 JDK 7

Sub Models C: > more conf 2. txt x: 0, 1 y: 2, 3 z: 4, 5 t: 6, 7 {x, y} @ 2 {z, t} @ 2 x, y: S 1 = {(0, 2), (0, 3), (1, 2), (1, 3)} z, t: S 2 = {(4, 6), (4, 7), (5, 6), (5, 7)} The solution for this model is the Cartesian product S 1 x. S 2

Sub Models C: >pict conf 2. txt x y z t 1 2 5 7 0 2 4 6 0 2 5 7 0 2 4 7 0 3 5 6 1 2 4 7 0 3 4 6 0 3 5 7 0 2 5 6 1 3 5 7 1 3 4 7 1 2 5 6 1 3 5 6 1 2 4 6 1 3 4 6 0 3 4 7

Defining a sub-model • In the model, express the pair (Application, Architecture, OS) as a sub-model • How many test cases are now generated ?

Trying to reduce the test case • Still too many test cases • Among the locales, we need to test EN above all. Other locales should be tested only once on only platform • Among the databases, we need to test DB 2 9. 7 above all. Other databases should be tested only once on any platform • Is it possible to express these constraints in PICT ?

Randomized generation • C: more try. txt x: 0, 1 y: 0, 1 z: a, b t: a, b {z, t} @1

Randomized generation C: >pict try 1. txt /r: 1 Used seed: 1 x y z t 1 0 a b 0 1 b a 1 1 b a 0 0 b a 0 1 a b C: >pict try 1. txt /r: 1000 Used seed: 1000 x y z t 0 1 b b 1 0 b b 0 0 a a 1 1 a a

Generating Test Cases • N-wise test cases can be generated with /o: N C: >pict mymodel. txt /o: 3 X Y Z 1 B 1 2 B 2 1 C 2 1 B 2 2 C 2 2 B 1 2 A 1 1 A 2 1 C 1 2 A 2 2 C 1

Further reading • More information on PICT: http: //msdn. microsoft. com/en-us/library/cc 150619. aspx • Many resources on pairwise testing: http: //www. pairwise. org/ The site is maintained by Jacek Czerwonka who developed PICT