Requirement Specifications based Test Automation Tao Xie University

Requirement Specifications based Test Automation Tao Xie University of Illinois at Urbana-Champaign http: //taoxie. cs. illinois. edu/courses/testing/ Work described in the slides was done in collaboration with the Pex team (Nikolai Tillmann, Peli de Halleux, Pratap Lakshman, et al. ) @Microsoft Research, students @Illinois ASE, and other collaborators

Techniques for writing tests □Black-box (from specifications) ◊ Equivalence partitioning ◊ Boundary value analysis □White-box (from code) ◊ Branch coverage □Fault-based testing (from common faults) 2

What is a Unit Test? A unit test is a small program with assertions. [Test. Method] public void Add() { Hash. Set set = new Hash. Set(); set. Add(3); set. Add(14); Assert. Are. Equal(set. Count, 2); } Many developers write such unit tests by hand, involving □ determining a meaningful sequence of method calls, □ selecting exemplary argument values (the test inputs), □ stating assertions. 3

Unit Testing: Measuring Quality □ Coverage: Are all parts of the program exercised? ◊ statements ◊ basic blocks ◊ explicit/implicit branches ◊… □ Assertions: Does the program do the right thing? ◊ test oracle Experience: □ Just high coverage or large number of assertions is no good quality indicator. □ Only both together are!

Advantages of tests as specs □Concrete, easy to understand □Don’t need new language □Easy to see if program meets the spec □Making tests forces you to talk to customer and learn the problem □Making tests forces you to think about design of system (classes, methods, etc. ) 5

Disadvantages of tests as specs □Too specific □Hard to test that something can’t happen ◊ Can’t withdraw more money than you have in the system ◊ Can’t break into the system ◊ Can’t cause a very long transaction that hangs the system □Tends to be verbose 6

Tests as specifications □Tests show to use the system □Tests need to be readable ◊ Need comments that describe their purpose or need good names ◊ Keep short, delete duplicate or redundant 7

Parameterized Unit Test □ A parameterized unit test is a small program that takes some inputs and states assumptions and assertions. Parameterized Unit Test See later slides on JUnit: @Theory (multiple parameters) @Parameters (single parameter) 8

Parameterized Unit Testing Parameterized Unit Tests □ serve as specifications □ can be leveraged by (automatic) test input generators □ fit in development environment, evolve with the code

Test Generation Process x. Unit Attributes Pex Attributes // Foo. Test. cs [Test. Class, Pex. Class] Partial Class partial class Foo. Test { [Pex. Method] Pex void Test(Foo foo) {…} Generated // Foo. Test. cs partial class Foo. Test { [Test. Method] void Test_1() { this. Test(new Foo(1)); } [Test. Method] void Test_1() { this. Test(new Foo(2)); } … Parameterized Unit Test Hand-written • User writes parameterized tests • Lives inside a test class } • Generated unit tests • Pex not required for re-execution • x. Unit unit tests http: //msdn. microsoft. com/en-us/library/wa 80 x 488(VS. 80). aspx 10

PUTs separate concerns PUTs separate two concerns: (1) The specification of external behavior (i. e. , assertions) (2) The selection of internal test inputs (i. e. , coverage) In many cases, a test generation tool (e. g. , Pex) can construct a small test suite with high coverage !

PUTs are algebraic specs A PUT can be read as a universally quantified, conditional axiom. int name, int data. name ≠ null ⋀ data ≠ null ⇒ equals( Read. Resource(name, Write. Resource(name, data)), data)

Pex 4 Fun – Turning Pex Online http: //pex 4 fun. com/default. aspx? language=CSharp&sample=_Template 1, 792, 222 clicked 'Ask Pex!'

Behind the Scene of Pex 4 Fun Dynamic Symbolic Execution (DSE) aka. Concolic Testing [Godefroid et al. 05][Sen et al. 05][Tillmann et al. 08] Instrument code to explore feasible paths http: //research. microsoft. com/pex/

http: //research. microsoft. com/pex/

Dynamic Symbolic Execution in Pex Choose next path Solve void Cover. Me(int[] a) { if (a == null) return; if (a. Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } F F a. Length>0 a==null T T Execute&Monitor Constraints to solve Input Observed constraints a!=null {} a==null a!=null && !(a. Length>0) a==null && a. Length>0 && a[0]!=1234567890 a==null && a. Length>0 && a[0]==1234567890 a!=null && a. Length>0 {0} a!=null && a. Length>0 && a[0]==123456890 {123…} Done: There is no path left. a[0]==123… F T http: //pex 4 fun. com/How. Does. Pex. Work

Pex is Part of Visual Studio 2015 Enterprise Edition! □As new feature of “Intelli. Test” Pex: 30 K downloads after 20 months Active user community: 1. 4 K forum posts during ~3 years https: //www. visualstudio. com/news/vs 2015 -vs#Testing

Impact of Intelli. Test NUnit Extension – 13024 downloads x. Unit. net Extension - 8589 downloads From http: //bbcodeplex. com/

Domain Matrix for Testing Complex Condition 19

Guide Pex to Generate Test Data http: //pex 4 fun. com/default. aspx? language=CSharp&sample=_Template [Pex. Method(Test. Emission. Filter = Pex. Test. Emission. Filter. All)] public void Test. Boundary. Values. And. Input. Partition(int x, int y) { //boundary values/partitions for x > 0 && x <= 10 && y >= 1 Pex. Assume. Is. True((x > 0)); if (x == 1) { } else if (x > 0) { } Pex. Assume. Is. True((x <= 10)); if (x == 10) { } else if (x <= 10) { } Pex. Assume. Is. True((y >= 1)); if (y == 1) { } else if (y > 1) { } using System; using Microsoft. Pex. Framework. Settings; [Pex. Class] public class Program { //insert your PUT here } } Details see http: //taoxie. cs. illinois. edu/publications/icsm 10 -coverage. pdf 20

Parameterized Unit Tests Supported by Pex/Pex 4 Fun using System; using Microsoft. Pex. Framework. Settings; [Pex. Class] public class Set { [Pex. Method] public static void test. Member. After. Insert. Not. Equal(Set s, int i, int j) { Pex. Assume. Is. True(s != null); Pex. Assume. Is. True(i != j); bool exist = s. member(i); s. insert(j); Pex. Assert. Is. True(exist); } …. } 21

Interface for Int. Set Class Int. Set { public Int. Set() {…}; public void insert(int e) { … } public Bool member(int e) { … } public void remove(int e) { … } } sort Int. Set imports Int, Bool signatures new : -> Int. Set insert : Int. Set × Int -> Int. Set member : Int. Set × Int -> Bool remove : Int. Set × Int -> Int. Set http: //www. cs. unc. edu/~stotts/723/adt. html 22

(Buggy) Implementation for Int. Set Class Int. Set { public Int. Set() {…}; public void insert(int e) { … } public Bool member(int e) { … } public void remove(int e) { … } } See the Set. cs that can be downloaded from http: //taoxie. cs. illinois. edu/courses/testing/Set. cs Let’s copy it to http: //pex 4 fun. com/default. aspx? language=CSharp&sample=_Template And Click “Ask Pex” 23

Parameterized Unit Tests Supported by Pex/Pex 4 Fun using System; using Microsoft. Pex. Framework. Settings; [Pex. Class] public class Set { } [Pex. Method] public static void test. Member. After. Insert. Not. Equal(Set s, int i, int j) { Pex. Assume. Is. True(s != null); Pex. Assume. Is. True(i != j); bool exist. Old = s. member(i); s. insert(j); bool exist = s. member(i); Pex. Assert. Is. True(exist. Old == exist); } …. Pex 4 Fun supports only one Pex. Method at a time; you can write multiple Pex. Methods but comment 24 out other lines of “[Pex. Method]” except one

Axioms for Int. Set variables i, j : Int; s : Int. Set Axioms: member(new(), i) = false member(insert(s, j), i) = if i = j then true else member(s, i) Is this complete? How do we know? http: //www. cs. unc. edu/~stotts/723/adt. html 25

Guidelines for Completeness □Classify methods: ◊ constructors: return Int. Set ◊ inspectors: take Int. Set as argument, returning some other value. □Identify key constructors, capable of constructing all possible object states ◊ e. g. , insert, new. □Identify others as auxiliary, ◊ e. g. , remove is a destructive constructor □Completeness requires (at least): ◊ every inspector/auxiliary constructor is defined by one equation for each key constructor. 26

Add More Axioms □remove(new(), i) = new() □remove(insert(s, j), i) = if i = j then remove(s, i) else insert(remove(s, i), j) Are we done yet? The completeness criterion (an equation defining member and remove for each of the new and insert constructors) is satisfied. 27

Guidelines for Completeness □But does this really specify sets? Do the following properties hold? □Order of insertion is irrelevant. ◊ insert(s, i), j) = insert(s, j), i) □Multiple insertion is irrelevant. ◊ insert(s, i) = insert(s, i) 28

Interface (Implementation) for UInt. Stack Class UInt. Stack { public UInt. Stack() {…}; public void Push(int k) { … } public void Pop() { … } public int Top() { … } public bool Is. Empty() { … } public int Max. Size() { … } public bool Is. Member(int k) { … } public bool Equals(UInt. Stack s) { … } public int Get. Number. Of. Elements() { … } public bool Is. Full() { … } } See the UInt. Stack. cs that can be downloaded from http: //taoxie. cs. illinois. edu/courses/testing/UInt. Stack. cs 29

Exercise: Write Parameterized Unit Tests (PUTs) Class UInt. Stack { Let’s copy it to http: //pex 4 fun. com/default. aspx? language= public UInt. Stack() {…}; public void Push(int k) { … } CSharp&sample=_Template And Click “Ask Pex” public void Pop() { … } public int Top() { … } Reminder: you have to public bool Is. Empty() { … } comment earlier written “[Pex. Method]” before public int Max. Size() { … } you try Pex on your public bool Is. Member(int k) { … } current PUT (Pex 4 Fun can handle only one public bool Equals(UInt. Stack s) { … } PUT at a time) public int Get. Number. Of. Elements() { … } public bool Is. Full() { … } } See the UInt. Stack. cs that can be downloaded from http: //taoxie. cs. illinois. edu/courses/testing/UInt. Stack. cs 30

One Sample PUT Below is a manually edited/created good factory method to guide Pex to generate various types of object states. Note that Pex also generates argument values for the factory method. [Pex. Method] public void Test. Push([Pex. Assume. Under. Test]UInt. Stack s, int i) { //UInt. Stack s = new UInt. Stack(); Pex. Assume. Is. True(!s. Is. Member(i)); int old. Count = s. Get. Number. Of. Elements(); s. Push(i); Pex. Assert. Is. True(s. Top() == i); Pex. Assert. Is. True(s. Get. Number. Of. Elements() == old. Count+1); Pex. Assert. Is. False(s. Is. Empty()); } 31

Guideline of Writing PUT • Setup: basic set up for invoking the method under test • Checkpoint: Run Pex to make sure that you don't • miss any Pex assumptions (preconditions) for the PUT Assert: add assertions for asserting behavior of the method under test, involving • Adding Pex assertions • Adding Pex assumptions for helping assert • Adding method sequences for helping assert

Setup • • • Select your method under test m Put its method call in your PUT Create a parameter for your PUT as the class under test c (annotated it with [Pex. Assume. Under. Test]) Create other parameters for your PUT for parameters of m if any Add Pex assumptions for preconditions for all these parameters of PUT if any

Setup - Example [Pex. Method] public void Test. Push([Pex. Assume. Under. Test]UInt. Stack s, int i) { s. Push(i); } You may write your factory method (see backup slides) to help Pex in test generation If you get exceptions thrown • if indicating program faults, fix them • If indicating lack of PUT assumptions, add PUT assumptions • If indicating insufficient factory method assumptions or inappropriate scenarios, add PUT assumptions or improve factory method.

Assert • Think about how you can assert the behavior • Do you need to invoke other (observer) helper • • methods in your assertions (besides asserting return values)? Do you need to add assumptions so that your assertions can be valid? Do you need to add some method sequence before the method under test to set up desirable state and cache values to be used in the assertions?

Targets for Asserting • • • Return value of the method under test (MUT) Argument object of MUT Receiver object properties being modified by MUT (if public fields, directly assertable) • How to assert them? • Think about the intended behavior! • If you couldn't do so easily, follow the guidelines discussed next

Cached Public Property Value • A property value before invoking MUT may need to be cached and later used. Pattern 2. 1/2. 2: Assume, Arrange, Act, Assert [Pex. Method] void Assume. Act. Assert(Array. List list, object item) { Pex. Assume. Is. Not. Null(list); // assume var count = list. Count; // arrange list. Add(item); // act Assert. Is. True(list. Count == count + 1); // assert }

Argument of MUT • Argument value of MUT may be used Pattern 2. 3: Constructor Test [Pex. Method] void Constructor(int capacity) { var list = new Array. List(capacity); // create Assert. Invariant(list); // assert invariant Assert. Are. Equal(capacity, list. Capacity); // assert }

Reciever or Argument of Earlier Method • Receiver or argument value of a method before invoking MUT value s parsed Pattern 2. 4/5: Roundtrip [Pex. Method] void To. String. Parse. Roundtrip(int value) { // two-way roundtrip string s = value. To. String(); int parsed = int. Parse(s); // assert Assert. Are. Equal(value, parsed); }

Observer Methods • Invoking observer methods on the modified object state Pattern 2. 6: State Relation [Pex. Method] void Insert. Contains(string value) { var list = new List<string>(); list. Add(value); Assert. Is. True(list. Contains(value)); } Each modified object property should be read by at least one observer method.

Observer Methods cont. • Forcing observer methods to return specific values (e. g. , true or false) can force you to add specific assumptions or scenarios [Pex. Method] void Push. Is. Full([Pex. Assume. Under. Test]UInt. Stack s, int value) { Pex. Assume. Is. True(s. Get. Size() == (s. Get. Max. Size()-1)); s. Push (value); Assert. Is. True(s. Is. Full ()); }

Parameterized Unit Tests in JUnit □ Problem: Testing A Function With Similar Values ◊ How To Avoid Test Code Bloat? □ Simple Example: Adding Two Numbers ◊ Adding a Given Pair of Numbers Is Just Like Adding Any Other Pair You Really Only Want to Write One Test ◊ □ Parameterized Unit Tests Call Constructor For Each Logical Set of Data Values ◊ Same Tests Are Then Run On Each Set of Data Values ◊ List of Data Values Identified with @Parameters Annotation 42 Introduction to Software Testing (Ch 1) © Ammann & Offutt

Parameterized Unit Tests in JUnit import org. junit. *; import org. junit. runner. Run. With; import org. junit. runners. Parameterized. Parameters; import static org. junit. Assert. *; import java. util. *; @Run. With(Parameterized. class) public class Param. Test { public int sum, a, b; public Param. Test (int sum, int a, int b) { this. sum = sum; this. a = a; this. b = b; } @Parameters public static Collection<Object[]> parameters() { return Arrays. as. List (new Object [][] {{0, 0, 0}, {2, 1, 1}}); } @Test public void addition. Test() { assert. Equals(sum, a+b); } } 43 Introduction to Software Testing (Ch 1) © Ammann & Offutt

JUnit Theories □ These Are Unit Tests With Actual Parameters ◊ So Far, We’ve Only Seen Parameterless Test Methods □ Contract Model: Assume, Act, Assert ◊ Assumptions (Preconditions) Limit Values Appropriately ◊ Action Performs Activity Under Scrutiny ◊ Assertions (Postconditions) Check Result @Theory public void remove. Then. Add. Does. Not. Change. Set( Set<String> set, String string) { // Parameters! assume. True(set. contains(string)) ; // Assume Set<String> copy = new Hash. Set<String>(set); // Act copy. remove(string); copy. add(string); assert. True (set. equals(copy)); // Assert // // System. out. println(“Instantiated test: “ + set + “, “ + string); } Introduction to Software Testing (Ch 1) © Ammann & Offutt 44

Question: Where Does Data Come From? □ Answer: ◊ All Combinations of Values from @Data. Point ◊ ◊ Annotations Where Assume Clause is True Four (of Nine) Combinations in This Particular Case Note: @Data. Point Format is an Array. @Data. Points public static String[] string = {"ant", "bat", "cat"}; @Data. Points public static Set[] sets = { new Hash. Set(Arrays. as. List("ant", "bat")), new Hash. Set(Arrays. as. List(“bat", “cat", “dog“, “elk”)), new Hash. Set(Arrays. as. List(“Snap”, “Crackle”, “Pop")) }; 45 Introduction to Software Testing (Ch 1) © Ammann & Offutt

JUnit Theories Need Boiler. Plate import org. junit. *; org. junit. runner. Run. With; static org. junit. Assert. *; static org. junit. Assume. *; import org. junit. experimental. theories. Data. Point; org. junit. experimental. theories. Data. Points; org. junit. experimental. theories. Theories; org. junit. experimental. theories. Theory; import java. util. *; @Run. With(Theories. class) public class Set. Theory. Test { … // See Earlier Slides } 46 Introduction to Software Testing (Ch 1) © Ammann & Offutt

Coding Duels t e r c se Pex computes “semantic diff” in cloud secret reference implementation vs. code written in browser You win when Pex finds no differences For more info, see our ICSE 2013 SEE paper: http: //taoxie. cs. illinois. edu/publications/icse 13 see-pex 4 fun. pdf

Behind the Scene of Pex for Fun behavior Secret Impl == Player Impl Secret Implementation class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } } 1, 594, 092 class Test { public static void Driver(int x) { if (Secret. Puzzle(x) != Player. Puzzle(x)) throw new Exception(“Mismatch”); } } Player Implementation class Player { public static int Puzzle(int x) { return x; } } 48

Code Hunt Programming Game Code Hunt has had over 450, 000 users in a year with around 1, 000 users a day https: //www. codehunt. com/ For more info, see our ICSE 2015 JSEET paper: http: //taoxie. cs. illinois. edu/publications/icse 15 jseet-codehunt. pdf

Modeling in Science and Engineering A model: □ Is an abstraction of the system from a particular perspective □ Supports investigation, construction and prediction □ Is not necessarily comprehensive □ Can be expressed as a table, graphical diagram, formal notation, etc. Ubiquitous and Universal Slides from https: //channel 9. msdn. com/Events/Tech. Ed/Europe/2009/DEV 316 ©W. Grieskamp 50

Software Modeling Styles and Notations Software Models Structure Class Diagram Component Diagram Behavior Interaction Based State Based Use Cases Guarded Update Machines (Code) Message Sequence Charts State Charts Traces/Patterns Slides from https: //channel 9. msdn. com/Events/Tech. Ed/Europe/2009/DEV 316 Focus of Spec Explorer ©W. Grieskamp 51

Model-Based Testing in a Nutshell Requirements Feedback Author Model Feedback Generate Issue Inputs (Test Sequences) Control Expected Outputs (Test Oracle) Observe Verdict Feedback Implementation Slides from https: //channel 9. msdn. com/Events/Tech. Ed/Europe/2009/DEV 316 ©W. Grieskamp

Model-Based Testing with Spec Explorer C# Model (or other. Net Language) Model Graph Explore Analyze Generate Test Suite Execute VSTT Result Slides from https: //channel 9. msdn. com/Events/Tech. Ed/Europe/2009/DEV 316 ©W. Grieskamp

Walkthrough: Modeling and Testing a Web Service □Example: A Simple Web Shop Bill Web Shop Client Add. To. Cart Checkout Billing Web Shop Service Ship System Under Test Slides from https: //channel 9. msdn. com/Events/Tech. Ed/Europe/2009/DEV 316 Shipping ©W. Grieskamp

Web Shop Implementation in Windows Communication Foundation (WCF) [Service. Contract(Callback. Contract = typeof(IWeb. Shop. Callback))] public interface IWeb. Shop { [Operation. Contract] // Add item to stock of web shop void Add. Item(string item. Name, int price); [Operation. Contract] // Add item to shopping cart of user bool Add. To. Cart(string user. Name, string item. Name); [Operation. Contract] // Check out void Checkout(string user. Name); } public interface IWeb. Shop. Callback { [Operation. Contract] // Initiate billing, returns true on success bool Bill(string user. Name, int amount); [Operation. Contract(Is. One. Way = true)] // Initiate shipping void Ship(string user. Name, string item. Name); } Slides from https: //channel 9. msdn. com/Events/Tech. Ed/Europe/2009/DEV 316 ©W. Grieskamp

Modeling Web Shop: Basic Approach □Import or declare action vocabulary in configuration script □Define state variables in C# model program □Define state update rules in C# model program □Define scenarios and generate state transition system Slides from https: //channel 9. msdn. com/Events/Tech. Ed/Europe/2009/DEV 316 ©W. Grieskamp

Modeling Web Shop: Action Vocabulary Adapter for the Reference to test System-Under-Test adapter method config Actions { action static void Test. Adapter. Add. Item(string item. Name, int price); action static void Test. Adapter. Add. To. Cart(string user. Name, string item. Name); action static void Test. Adapter. Checkout(string user. Name); action event static void Test. Adapter. Billed(string user. Name, int amount, bool success); action event static void Test. Adapter. Shipped(string user. Name, string item. Name); } Indicates that this action is an event raised by adapter Slides from https: //channel 9. msdn. com/Events/Tech. Ed/Europe/2009/DEV 316 ©W. Grieskamp

Modeling Web. Shop: Define Model State in C# enum Shopper. Status { Shopping, Checked. Out, Billed } class Shopper. Info { // Current status of this user internal Shopper. Status; // Items on cart so far internal Sequence. Container<string> Items; } // Foreach user shopping, contains info static Map. Container<string, Shopper. Info> shoppers; // Foreach item in the shop, contains price static Map. Container<string, int> items; Slides from https: //channel 9. msdn. com/Events/Tech. Ed/Europe/2009/DEV 316 ©W. Grieskamp

Modeling Web. Shop: Defining State Update Rules [Action(“Add. To. Cart(user. Name, item. Name)”)] static void Add. To. Cart(string user. Name, string item. Name) { Contracts. Requires(items. Contains. Key(item. Name)); if (!shoppers. Contains. Key(user. Name)) shoppers[user. Name] = new Shopper. Info(); } Binding rule to action in action vocabulary Contracts. Requires(In. Status(user. Name, Shopper. Status. Shopping)); shoppers[user. Name]. Items. Add(item. Name); Condition under which rule is static bool In. Status(string user. Name, Shopper. Status enabled. status) Fires a { state update } return shoppers. Contains. Key(user. Name) && shoppers[user. Name]. Status == status; State update Slides from https: //channel 9. msdn. com/Events/Tech. Ed/Europe/2009/DEV 316 ©W. Grieskamp

Modeling Web. Shop: Define Scenarios machine Multiple. Item. Scenario() : Config { Add. Item("Bread", 2); Add. Item("Cheese", 5); Add. Item("Wine", 10) Add. To. Cart("Paul", "Wine"); Add. To. Cart("Paul", "Cheese"); (Checkout | Billed | Shipped)* } Sequence of actions Or’ing actions in repetition (Note parameters omitted) machine Multiple. Item. Slice() : Config { Multiple. Item. Scenario || construct model program from Config } Combining scenario with C# model program Slides from https: //channel 9. msdn. com/Events/Tech. Ed/Europe/2009/DEV 316 ©W. Grieskamp

Modeling Web. Shop: Exploration of the Scenario Parameters and control flow undetermined Slides from https: //channel 9. msdn. com/Events/Tech. Ed/Europe/2009/DEV 316 ©W. Grieskamp

Modeling Web. Shop: Exploration of the Model Slice Choice point for system -under-test Slides from https: //channel 9. msdn. com/Events/Tech. Ed/Europe/2009/DEV 316 ©W. Grieskamp

Modeling Web. Shop: Execution of generated test code Model expects bill of $2, implementation bills $4 Slides from https: //channel 9. msdn. com/Events/Tech. Ed/Europe/2009/DEV 316 ©W. Grieskamp

“Blue. Line” Technical Document Testing Project □ 222 open protocol specifications tested ◊ 22, 847 pages □ 66, 962 person days (250+ years) □Vendor test teams ◊ Hyderabad: 250 ◊ Beijing: 100 □Tool dev center in Beijing (Spec Explorer and other) □~10, 000 document bugs filed Slides from https: //channel 9. msdn. com/Events/Tech. Ed/Europe/2009/DEV 316 ©W. Grieskamp

Comparison of Model-Based/Traditional Testing □Vendor teams had (guided) choice which approach to use □Out of 222 protocols approx. ½ modeled, ½ traditional □Modeling resulted in 42% productivity gain in average (some teams reached more than 300%) ◊ Overall, modeling saved 50 person-years (12, 547 days) □No special education background was required ◊ Teams of junior engineers ◊ Mostly vendors, fresh out of college Slides from https: //channel 9. msdn. com/Events/Tech. Ed/Europe/2009/DEV 316 ©W. Grieskamp

Summary Parameterized Unit Test Intelli. Test http: //research. microsoft. com/pex/ NModel A Model-Based Testing tool from Microsoft https: //msdn. microsoft. com/en-us/library/ee 620411. aspx https: //visualstudiogallery. msdn. microsoft. com/271 d 0904 - http: //nmodel. codeplex. com/ f 178 -4 ce 9 -956 b-d 9 bfa 4902745

Q&A 67

Code Hunt Programming Game

69

Test-Driven Development (TDD) □Basic Idea: ◊ Write tests before code ◊ Refine code with new tests □In more detail, TDD is a cycle of steps: ◊ Add a test, ◊ Run it and watch it fail, ◊ Change the code as little as possible such that the test should pass, ◊ Run the test again and see it succeed, ◊ Refactor the code if needed.

Note: TDD and specifications □TDD encourages writing specifications before code ◊ Exemplary specification □Later, we will generalize TDD to Parameterized TDD ◊ Axiomatic specifications

Parameterized Test. Driven Development Write/refine Contract as PUT Bug in PUT Write/refine Code of Implementation Bug in Code Run Pex no failures Use Generated Tests for Regression failures Fix-it (with Pex), Debug with generated tests

Code Hunt Programming Game

Code Hunt Programming Game

Code Hunt Programming Game

Code Hunt Programming Game

Code Hunt Programming Game

Code Hunt Programming Game

Code Hunt Programming Game

Code Hunt Programming Game

Code Hunt Programming Game

Code Hunt Programming Game

Code Hunt Programming Game

It’s a game! iterative gameplay adaptive ret c personalized se no cheating clear winning criterion code test cases

(Buggy) Implementation for Int. Set Class Int. Set { public Int. Set() {…}; public void insert(int e) { … } public Bool member(int e) { … } public void remove(int e) { … } } See the Set. cs that can be downloaded from http: //taoxie. cs. illinois. edu/courses/testing/Set. cs Let’s copy it to http: //pex 4 fun. com/default. aspx? language=CSharp&sample=_Template And Click “Ask Pex” 85

Recall: Parameterized Unit Tests Supported by Pex/Pex 4 Fun using System; using Microsoft. Pex. Framework. Settings; [Pex. Class] public class Set { [Pex. Method] public static void test. Member. After. Insert. Not. Equal(Set s, int i, int j) { Pex. Assume. Is. True(s != null); Pex. Assume. Is. True(i != j); bool exist. Old = s. member(i); s. insert(j); bool exist = s. member(i); Pex. Assert. Is. True(exist. Old == exist); } …. } 86

Force Pex/Pex 4 Fun to Display All Explored Test Inputs/Paths using System; using Microsoft. Pex. Framework. Settings; [Pex. Class] public class Set { [Pex. Method(Test. Emission. Filter=Pex. Test. Emission. Filter. All)] public static void test. Member. After. Insert. Not. Equal(Set s, int i, int j) { Pex. Assume. Is. True(s != null); Pex. Assume. Is. True(i != j); bool exist = s. member(i); s. insert(j); Pex. Assert. Is. True(exist); } …. } 87

Factory Method: Help Pex Generate Desirable Object States In class, we show the factory method as below automatically synthesized by Pex after a user clicks “ 1 Object Creation” issue and then click “Accept/Edit Factory Method”. But it is not good enough to generate various types of object states. [Pex. Factory. Method(typeof(UInt. Stack))] public static UInt. Stack Create(int k_i) { UInt. Stack u. Int. Stack = new UInt. Stack(); u. Int. Stack. Push(k_i); return u. Int. Stack; // TODO: Edit factory method of UInt. Stack // This method should be able to configure the object in all possible ways. // Add as many parameters as needed, // and assign their values to each field by using the API. } 88

Factory Method: Help Pex Generate Desirable Object States Below is a manually edited/created good factory method to guide Pex to generate various types of object states. Note that Pex also generates argument values for the factory method. [Pex. Factory. Method(typeof(UInt. Stack))] public static UInt. Stack Create. Varied. Size. Any. Elems. Stack(int[] elems) { Pex. Assume. Is. Not. Null(elems); UInt. Stack s = new UInt. Stack(); Pex. Assume. Is. True(elems. Length <= (s. Max. Size() + 1)); for (int i = 0; i < elems. Length; i++) s. Push(elems[i]); return s; } 89

Pex 4 Fun Not Supporting Factory Method - Workaround If you try PUTs on Pex 4 Fun, which doesn’t support factory method, you can “embed” the factory method like the highlighted code portion below [Pex. Method] public void Test. Push(int[] elems, int i) { Pex. Assume. Is. Not. Null(elems); UInt. Stack s = new UInt. Stack(); Pex. Assume. Is. True(elems. Length <= (s. Max. Size() + 1)); for (int i = 0; i < elems. Length; i++) s. Push(elems[i]); //UInt. Stack s = new UInt. Stack(); Pex. Assume. Is. True(!s. Is. Member(i)); int old. Count = s. Get. Number. Of. Elements(); s. Push(i); Pex. Assert. Is. True(s. Top() == i); Pex. Assert. Is. True(s. Get. Number. Of. Elements() == old. Count+1); Pex. Assert. Is. False(s. Is. Empty()); } 90

Understanding the Web Shop Which of the following traces is valid? (assuming an initialization of the shop with Add. Item(“wine”, 10) and Add. Item(“cheese”, 5)) 1 Add. To. Cart(“Peter”, “wine”) Checkout(“Peter”) Bill(“Peter”, 10)/true Ship(“Peter”, ”wine”) 2 Add. To. Cart(“Peter”, “wine”) Checkout(“Peter”) Bill(“Peter”, 10)/false Ship(“Peter”, ”wine”) 3 Add. To. Cart(“Peter”, “wine”) Checkout(“Peter”) Bill(“Peter”, 12)/true Ship(“Peter”, ”wine”) 4 Add. To. Cart(“Peter”, “wine”) Add. To. Cart(“Peter”, ”cheese”) Checkout(“Peter”) Bill(“Peter”, 15)/true Ship(“Peter”, ”wine”) Ship(“Peter”, ”cheese”) 5 Add. To. Cart(“Peter”, “wine”) Add. To. Cart(“Peter”, ”cheese”) Checkout(“Peter”) Bill(“Peter”, 15)/true Ship(“Peter”, ”cheese”) Ship(“Peter”, ”wine”) Slides from https: //channel 9. msdn. com/Events/Tech. Ed/Europe/2009/DEV 316 ©W. Grieskamp