Advances in Unit Testing Theory and Practice Tao

Advances in Unit Testing: Theory and Practice Tao Xie 1 Nikolai Tillmann 2 Pratap Lakshman 2 1 University of Illinois at Urbana-Champaign 2 Microsoft Tools for Software Engineers/Developer Division Materials: http: //taoxie. cs. illinois. edu/courses/testing/

What is a Unit Test? A unit test is a small program with assertions. [Test. Method] public void Add(Hash. Set s, int x, int y) { Assume. Is. True(!s. Is. Member(x)); int old. Count = s. Get. Size(); s. Add(x); Assert. Are. Equal(set. Count, old. Count + 1); } Many developers write such unit tests by hand. This involves � determining a meaningful sequence of method calls, � selecting exemplary argument values (the test inputs), � stating assertions. 2

Unit Testing: Benefits �Design and specification by example �Code coverage and regression testing confidence in correctness preserving behavior �Short feedback loop unit tests exercise little code failures are easy to debug �Documentation

Unit Testing: Measuring Quality � Coverage: Are all parts of the program exercised? statements basic blocks explicit/implicit branches … � Assertions: Does the program do the right thing? test oracle Experience: � Just high coverage or large number of assertions is no good quality indicator. � Only both together are!

Advantages of tests as specs �Concrete, easy to understand �Don’t need new language �Easy to see if program meets the spec �Making tests forces you to talk to customer and learn the problem �Making tests forces you to think about design of system (classes, methods, etc. ) 5

Disadvantages of tests as specs �Too specific �Hard to test that something can’t happen Can’t withdraw more money than you have in the system Can’t break into the system Can’t cause a very long transaction that hangs the system �Tends to be verbose 6

Parameterized Unit Test � A parameterized unit test is a small program that takes some inputs and states assumptions and assertions. Parameterized Unit Test 7

Parameterized Unit Testing Parameterized Unit Tests � serve as specifications � can be leveraged by (automatic) test input generators � fit in development environment, evolve with the code

Test Generation Process x. Unit Attributes Pex Attributes // Foo. Test. cs [Test. Class, Pex. Class] partial class Foo. Test { [Pex. Method] void Test(Foo foo) {…} Generated Partial Class Parameterized Unit Test Hand-written • User writes parameterized tests • Lives inside a test class Pex // Foo. Test. cs partial class Foo. Test { [Test. Method] void Test_1() { this. Test(new Foo(1)); } [Test. Method] void Test_1() { this. Test(new Foo(2)); } … } • Generated unit tests • Pex not required for re-execution • x. Unit unit tests http: //msdn. microsoft. com/en-us/library/wa 80 x 488(VS. 80). aspx 9

PUTs separate concerns PUTs separate two concerns: (1) The specification of external behavior (i. e. , assertions) (2) The selection of internal test inputs (i. e. , coverage) In many cases, a test generation tool (e. g. , Pex) can construct a small test suite with high coverage !

PUTs are algebraic specs A PUT can be read as a universally quantified, conditional axiom. int name, int data. name ≠ null ⋀ data ≠ null ⇒ equals( Read. Resource(name, Write. Resource(name, data)), data)

Parameterized Unit Tests in JUnit import org. junit. *; import org. junit. runner. Run. With; import org. junit. runners. Parameterized. Parameters; import static org. junit. Assert. *; import java. util. *; @Run. With(Parameterized. class) public class Param. Test { public int sum, a, b; public Param. Test (int sum, int a, int b) { this. sum = sum; this. a = a; this. b = b; } @Parameters public static Collection<Object[]> parameters() { return Arrays. as. List (new Object [][] {{0, 0, 0}, {2, 1, 1}}); } @Test public void addition. Test() { assert. Equals(sum, a+b); } } 12

JUnit Theories � These Are Unit Tests With Actual Parameters So Far, We’ve Only Seen Parameterless Test Methods � Contract Model: Assume, Act, Assert Assumptions (Preconditions) Limit Values Appropriately Action Performs Activity Under Scrutiny Assertions (Postconditions) Check Result @Theory public void remove. Then. Add. Does. Not. Change. Set( Set<String> set, String string) { // Parameters! assume. True(set. contains(string)) ; // Assume Set<String> copy = new Hash. Set<String>(set); // Act copy. remove(string); copy. add(string); assert. True (set. equals(copy)); // Assert // // System. out. println(“Instantiated test: “ + set + “, “ + string); } 13

Question: Where Does Data Come From? Answer: All Combinations of Values from @Data. Point Annotations Where Assume Clause is True Four (of Nine) Combinations in This Particular Case Note: @Data. Point Format is an Array. @Data. Points public static String[] string = {"ant", "bat", "cat"}; @Data. Points public static Set[] sets = { new Hash. Set(Arrays. as. List("ant", "bat")), new Hash. Set(Arrays. as. List(“bat", “cat", “dog“, “elk”)), new Hash. Set(Arrays. as. List(“Snap”, “Crackle”, “Pop")) }; 14

JUnit Theories Need Boiler. Plate import org. junit. *; org. junit. runner. Run. With; static org. junit. Assert. *; static org. junit. Assume. *; import org. junit. experimental. theories. Data. Point; org. junit. experimental. theories. Data. Points; org. junit. experimental. theories. Theories; org. junit. experimental. theories. Theory; import java. util. *; @Run. With(Theories. class) public class Set. Theory. Test { … // See Earlier Slides } 15

(Automated) Test Generation �Human Expensive, incomplete, … �Brute Force Pairwise, predefined data, etc… �Tool Automation!!

State-of-the-Art/Practice Test Generation Tools Running Symbolic Path. Finder. . . … ====================== results no errors detected ====================== statistics elapsed time: 0: 02 states: new=4, visited=0, backtracked=4, end=2 search: max. Depth=3, constraints=0 choice generators: thread=1, data=2 heap: gc=3, new=271, free=22 instructions: 2875 max memory: 81 MB loaded code: classes=71, methods=884 … 17

http: //research. microsoft. com/pex/

Pex is Part of Visual Studio 2015 Enterprise Edition! �As new feature of “Intelli. Test” https: //www. visualstudio. com/news/vs 2015 -vs#Testing

Pex 4 Fun – Turning Pex Online http: //pex 4 fun. com/default. aspx? language=CSharp&sample=_Template 1, 767, 012 clicked 'Ask Pex!'

Code Hunt Programming Game https: //www. codehunt. com/ 130, 000 users as of March 2015 after 1 year launch

What are Behind Pex �NOT Random: Cheap, Fast “It passed a thousand tests” feeling �… �But Dynamic Symbolic Execution: e. g. , Pex, CUTE, EXE White box Constraint Solving

Behind the Scene of Pex Dynamic Symbolic Execution (DSE) aka. Concolic Testing [Godefroid et al. 05][Sen et al. 05][Tillmann et al. 08] Instrument code to explore feasible paths http: //research. microsoft. com/pex/

Dynamic Symbolic Execution Choose next path Code to generate inputs for: void Cover. Me(int[] a) { if (a == null) return; if (a. Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } F F a. Length>0 a==null T Solve Constraints to solve null a==null a!=null && !(a. Length>0) a!=null && a. Length>0 && a[0]!=1234567890 {123…} a!=null && a. Length>0 && a[0]==1234567890 Done: There is no path left. a[0]==123… F Observed constraints {0} condition Negated a!=null && a. Length>0 && a[0]==1234567890 T Data {} a!=null && a. Length>0 Execute&Monitor T

Explosion of Search Space There are decision procedures for individual path conditions, but… �Number of potential paths grows exponentially with number of branches �Reachable code not known initially �Without guidance, same loop might be unfolded forever Fitnex search strategy [Xie et al. DSN 09] http: //taoxie. cs. illinois. edu/publications/dsn 09 -fitnex. pdf

Successful Case of MSR Testing Tool: Pex & Relatives �Pex (released on May 2008) Shipped with Visual Studio 15 as Intelli. Test 30, 388 download# (20 months, Feb 08 -Oct 09) A Journey of Bringing Active user community: 1, 436 forum posts during ~3 years (Oct 08 - Nov 11) Automated Unit Test �Moles (released on Sept 2009) Generation to Practice Shipped with Visual Studio 12 as Fakes 22, 466 download# (10 months, Apr 13 -Jan 14): Code Digger “Provide Microsoft Fakes w/ all Visual Studio editions” got 1, 457 community votes

Lesson 1. Started as (Evolved) Dream Moles/Fakes Parameterized Unit Tests Supported by Pex void Test. Add(Array. List a, object o) { Assume. Is. True(a!=null); int i = a. Count; a. Add(o); Assert. Is. True(a[i] == o); } Code Digger Pex 4 Fun/Code Hunt � Surrounding (Moles/Fakes) � Simplifying (Code Digger) � Retargeting (Pex 4 Fun/Code Hunt)

Lesson 2. Chicken and Egg Macro Perspective Tool Adoption by (Mass) Target Users Tool Shipping with Visual Studio Micro Perspective � Developer/manager: “Who is using your tool? ” � Pex team: “Do you want to be the first? ” � Developer/manager: “I love your tool but no. ”

Lesson 3. Human Factors – Generated Data Consumed by Human � Developer: “Code digger generates a lot of “” strings as input. I can’t find a way to create such a string via my own C# code. Could any one show me a C# snippet? I meant zero terminated string. ” � Pex team: “In C#, a in a string does not mean zerotermination. It’s just yet another character in the string (a very simple character where all bits are zero), and you can create as Pex shows the value: “”. ” � Developer: “Your tool generated “”” � Pex team: “What did you expect? ” � Developer: “Marc. ”

Lesson 3. Human Factors – Generated Name Consumed by Human � Developer: “Your tool generated a test called Foo 001. I don’t like it. ” � Pex team: “What did you expect? ” � Developer: “Foo_Should_Fail_When_Bar_Is_Negative. ”

Lesson 3. Human Factors – Generated Results Consumed by Human Object Creation messages suppressed (related to Covana by Xiao et al. [ICSE’ 11]) http: //taoxie. cs. illinois. edu/publications/icse 11 -covana. pdf Exception Tree View Exploration Results View

Lesson 4. Best vs. Worst Cases public bool Test. Loop(int x, int[] y) { if (x == 90) { Fitnex by Xie et al. [DSN’ 09] for (int i = 0; i < y. Length; i++) if (y[i] == 15) x++; Key observations: with respect to the if (x == 110) coverage target return true; not all paths are equally promising for } branch-node flipping return false; } not all branch nodes are equally To avoid local optimal or biases, the fitness-guided strategy is integrated with Pex’s fairness search strategies • Our solution: promising to flip – Prefer to flip branch nodes on the most promising paths – Prefer to flip the most promising branch nodes on paths – Fitness function to measure “promising” extents http: //taoxie. cs. illinois. edu/publications/dsn 09 -fitnex. pdf

Lesson 5. Tool Users’ Stereotypical Mindset or Habits � “Simply one mouse click and then everything would work just perfectly” Often need environment isolation w/ Moles/Fakes or factory methods, … � “One mouse click, a test generation tool would detect all or most kinds of faults in the code under test” Developer: “Your tool only finds null references. ” Pex team: “Did you write any assertions? ” Developer: “Assertion? ? ? ” � “I do not need test generation; I already practice unit testing (and/or TDD). Test generation does not fit into the TDD process”

Lesson 6. Practitioners’ Voice Gathered feedback from target tool users � Directly, e. g. , via MSDN Pex forum, tech support, outreach to MS engineers and . NET user groups � Indirectly, e. g. , via interactions with MS Visual Studio team (a tool vendor to its huge user base) �Motivations of Moles Refactoring testability issue faced resistance in practice Observation at Agile 2008: high attention on mock objects and tool supports

Lesson 7. Collaboration w/ Academia � Win-win collaboration model Win (Ind Lab): longer-term research innovation, man power, research impacts, … Win (Univ): powerful infrastructure, relevant/important problems in practice, both research and industry impacts, … �Industry-located Collaborations Faculty visits, e. g. , Fitnex, Pex 4 Fun Student internships, e. g. , Flo. PSy, Dy. Gen, state cov �Academia-located Collaborations

Lesson 7. Collaboration w/ Academia-located Collaborations �Immediate indirect impacts, e. g. , Reggae [ASE’ 09 s] Rex MSeq. Gen [FSE’ 09] Dy. Gen Guided Cov [ICSM’ 10] state coverage �Long-term indirect impacts, e. g. , Dy. Sy by Csallner et al. [ICSE’ 08] Seeker [OOPSLA’ 11] Covana [ICSE’ 11]

Summary of Lessons Learned �Pex practice impacts Moles/Fakes, Code Digger, Pex 4 Fun/Code Hunt �Lessons in transferring tools Started as (Evolved) Dream Chicken and Egg Human Factors Best vs. Worst Cases Tool Users’ Stereotypical Mindset or Habits Practitioners’ Voice Collaboration w/ Academia

Experience Reports on Successful Tool Transfer Nikolai Tillmann, Jonathan de Halleux, and Tao Xie. Transferring an Automated Test Generation Tool to Practice: From Pex to Fakes and Code Digger. In Proceedings of ASE 2014, Experience Papers. http: //taoxie. cs. illinois. edu/publications/ase 14 -pexexperiences. pdf � Jian-Guang Lou, Qingwei Lin, Rui Ding, Qiang Fu, Dongmei Zhang, and Tao Xie. Software Analytics for Incident Management of Online Services: An Experience Report. In Proceedings ASE 2013, Experience Paper. http: //taoxie. cs. illinois. edu/publications/ase 13 -sas. pdf � Dongmei Zhang, Shi Han, Yingnong Dang, Jian-Guang Lou, Haidong Zhang, and Tao Xie. Software Analytics in Practice. IEEE Software, Special Issue on the Many Faces of Software Analytics, 2013. http: //taoxie. cs. illinois. edu/publications/ieeesoft 13 -softanalytics. pdf � Yingnong Dang, Dongmei Zhang, Song Ge, Chengyun Chu, Yingjun Qiu, and Tao Xie. XIAO: Tuning Code Clones at Hands of Engineers in Practice. In Proceedings of ACSAC 2012. http: //taoxie. cs. illinois. edu/publications/acsac 12 -xiao. pdf �

Parameterized Unit Tests Supported by Pex/Pex 4 Fun using System; using Microsoft. Pex. Framework. Settings; [Pex. Class] public class Set { [Pex. Method] public static void test. Member. After. Insert. Not. Equal(Set s, int i, int j) { Pex. Assume. Is. True(s != null); Pex. Assume. Is. True(i != j); bool exist = s. member(i); s. insert(j); Pex. Assert. Is. True(exist); } …. } 39

Interface for Int. Set Class Int. Set { public Int. Set() {…}; public void insert(int e) { … } public Bool member(int e) { … } public void remove(int e) { … } } sort Int. Set imports Int, Bool signatures new : -> Int. Set insert : Int. Set × Int -> Int. Set member : Int. Set × Int -> Bool remove : Int. Set × Int -> Int. Set http: //www. cs. unc. edu/~stotts/723/adt. html 40

(Buggy) Implementation for Int. Set Class Int. Set { public Int. Set() {…}; public void insert(int e) { … } public Bool member(int e) { … } public void remove(int e) { … } } See the Set. cs that can be downloaded from http: //taoxie. cs. illinois. edu/courses/testing/Set. cs Let’s copy it to http: //pex 4 fun. com/default. aspx? language=CSharp&sample=_Template And Click “Ask Pex” 41

Parameterized Unit Tests Supported by Pex/Pex 4 Fun using System; using Microsoft. Pex. Framework. Settings; [Pex. Class] public class Set { [Pex. Method] public static void test. Member. After. Insert. Not. Equal(Set s, int i, int j) { Pex. Assume. Is. True(s != null); Pex. Assume. Is. True(i != j); bool exist. Old = s. member(i); s. insert(j); bool exist = s. member(i); Pex. Assert. Is. True(exist. Old == exist); } …. } Pex 4 Fun supports only one Pex. Method at a time; you can write multiple Pex. Methods but comment out other lines of “[Pex. Method]” except one 42

Axioms for Int. Set variables i, j : Int; s : Int. Set Axioms: member(new(), i) = false member(insert(s, j), i) = if i = j then true else member(s, i) Is this complete? How do we know? http: //www. cs. unc. edu/~stotts/723/adt. html 43

Guidelines for Completeness �Classify methods: constructors: return Int. Set inspectors: take Int. Set as argument, returning some other value. �Identify key constructors, capable of constructing all possible object states e. g. , insert, new. �Identify others as auxiliary, e. g. , remove is a destructive constructor �Completeness requires (at least): every inspector/auxiliary constructor is defined by one equation for each key constructor. 44

Add More Axioms �remove(new(), i) = new() �remove(insert(s, j), i) = if i = j then remove(s, i) else insert(remove(s, i), j) Are we done yet? The completeness criterion (an equation defining member and remove for each of the new and insert constructors) is satisfied. 45

Guidelines for Completeness �But does this really specify sets? Do the following properties hold? �Order of insertion is irrelevant. insert(s, i), j) = insert(s, j), i) �Multiple insertion is irrelevant. insert(s, i) = insert(s, i) 46

Interface (Implementation) for UInt. Stack Class UInt. Stack { public UInt. Stack() {…}; public void Push(int k) { … } public void Pop() { … } public int Top() { … } public bool Is. Empty() { … } public int Max. Size() { … } public bool Is. Member(int k) { … } public bool Equals(UInt. Stack s) { … } public int Get. Number. Of. Elements() { … } public bool Is. Full() { … } } See the UInt. Stack. cs that can be downloaded from http: //taoxie. cs. illinois. edu/courses/testing/UInt. Stack. cs 47

Take-Home Exercise: Write Parameterized Unit Tests (PUTs) Class UInt. Stack { Let’s copy it to http: //pex 4 fun. com/default. aspx? language=C public UInt. Stack() {…}; Sharp&sample=_Template public void Push(int k) { … } And Click “Ask Pex” public void Pop() { … } public int Top() { … } Reminder: you have to public bool Is. Empty() { … } comment earlier written public int Max. Size() { … } “[Pex. Method]” before you try Pex on your current public bool Is. Member(int k) { … } PUT (Pex 4 Fun can handle public bool Equals(UInt. Stack s) { … } only one PUT at a time) public int Get. Number. Of. Elements() { … } public bool Is. Full() { … } } See the UInt. Stack. cs that can be downloaded from http: //taoxie. cs. illinois. edu/courses/testing/UInt. Stack. cs 48

Recall: Parameterized Unit Tests Supported by Pex/Pex 4 Fun using System; using Microsoft. Pex. Framework. Settings; [Pex. Class] public class Set { [Pex. Method] public static void test. Member. After. Insert. Not. Equal(Set s, int i, int j) { Pex. Assume. Is. True(s != null); Pex. Assume. Is. True(i != j); bool exist. Old = s. member(i); s. insert(j); bool exist = s. member(i); Pex. Assert. Is. True(exist. Old == exist); } …. } 49

Force Pex/Pex 4 Fun to Display All Explored Test Inputs/Paths using System; using Microsoft. Pex. Framework. Settings; [Pex. Class] public class Set { [Pex. Method(Test. Emission. Filter=Pex. Test. Emission. Filter. All)] public static void test. Member. After. Insert. Not. Equal(Set s, int i, int j) { Pex. Assume. Is. True(s != null); Pex. Assume. Is. True(i != j); bool exist = s. member(i); s. insert(j); Pex. Assert. Is. True(exist); } …. } 50

Factory Method: Help Pex Generate Desirable Object States In class, we show the factory method as below automatically synthesized by Pex after a user clicks “ 1 Object Creation” issue and then click “Accept/Edit Factory Method”. But it is not good enough to generate various types of object states. [Pex. Factory. Method(typeof(UInt. Stack))] public static UInt. Stack Create(int k_i) { UInt. Stack u. Int. Stack = new UInt. Stack(); u. Int. Stack. Push(k_i); return u. Int. Stack; // TODO: Edit factory method of UInt. Stack // This method should be able to configure the object in all possible ways. // Add as many parameters as needed, // and assign their values to each field by using the API. } 51

Factory Method: Help Pex Generate Desirable Object States Below is a manually edited/created good factory method to guide Pex to generate various types of object states. Note that Pex also generates argument values for the factory method. [Pex. Factory. Method(typeof(UInt. Stack))] public static UInt. Stack Create. Varied. Size. Any. Elems. Stack(int[] elems) { Pex. Assume. Is. Not. Null(elems); UInt. Stack s = new UInt. Stack(); Pex. Assume. Is. True(elems. Length <= (s. Max. Size() + 1)); for (int i = 0; i < elems. Length; i++) s. Push(elems[i]); return s; } 52

One Sample PUT Below is a manually edited/created good factory method to guide Pex to generate various types of object states. Note that Pex also generates argument values for the factory method. [Pex. Method] public void Test. Push([Pex. Assume. Under. Test]UInt. Stack s, int i) { //UInt. Stack s = new UInt. Stack(); Pex. Assume. Is. True(!s. Is. Member(i)); int old. Count = s. Get. Number. Of. Elements(); s. Push(i); Pex. Assert. Is. True(s. Top() == i); Pex. Assert. Is. True(s. Get. Number. Of. Elements() == old. Count+1); Pex. Assert. Is. False(s. Is. Empty()); } 53

Pex 4 Fun Not Supporting Factory Method - Workaround If you try PUTs on Pex 4 Fun, which doesn’t support factory method, you can “embed” the factory method like the highlighted code portion below [Pex. Method] public void Test. Push(int[] elems, int i) { Pex. Assume. Is. Not. Null(elems); UInt. Stack s = new UInt. Stack(); Pex. Assume. Is. True(elems. Length <= (s. Max. Size() + 1)); for (int i = 0; i < elems. Length; i++) s. Push(elems[i]); //UInt. Stack s = new UInt. Stack(); Pex. Assume. Is. True(!s. Is. Member(i)); int old. Count = s. Get. Number. Of. Elements(); s. Push(i); Pex. Assert. Is. True(s. Top() == i); Pex. Assert. Is. True(s. Get. Number. Of. Elements() == old. Count+1); Pex. Assert. Is. False(s. Is. Empty()); } 54

Guideline of Writing PUT Setup: basic set up for invoking the method under test • • Checkpoint: Run Pex to make sure that you don't miss any Pex assumptions (preconditions) for the PUT Assert: add assertions for asserting behavior of the method under test, involving • • Adding Pex assertions Adding Pex assumptions for helping assert Adding method sequences for helping assert

Setup • • • Select your method under test m Put its method call in your PUT Create a parameter for your PUT as the class under test c (annotated it with [Pex. Assume. Under. Test]) Create other parameters for your PUT for parameters of m if any Add Pex assumptions for preconditions for all these parameters of PUT if any

Setup - Example [Pex. Method] public void Test. Push([Pex. Assume. Under. Test]UInt. Stack s, int i) { s. Push(i); } You may write your factory method to help Pex in test generation If you get exceptions thrown • if indicating program faults, fix them • If indicating lack of PUT assumptions, add PUT assumptions • If indicating insufficient factory method assumptions or inappropriate scenarios, add PUT assumptions or improve factory method.

Assert Think about how you can assert the behavior • • Do you need to invoke other (observer) helper methods in your assertions (besides asserting return values)? Do you need to add assumptions so that your assertions can be valid? Do you need to add some method sequence before the method under test to set up desirable state and cache values to be used in the assertions?

Targets for Asserting • • • Return value of the method under test (MUT) Argument object of MUT Receiver object properties being modified by MUT (if public fields, directly assertable) • How to assert them? • • Think about the intended behavior! If you couldn't do so easily, follow the guidelines discussed next

Cached Public Property Value • A property value before invoking MUT may need to be cached and later used. Pattern 2. 1/2. 2: Assume, Arrange, Act, Assert [Pex. Method] void Assume. Act. Assert(Array. List list, object item) { Pex. Assume. Is. Not. Null(list); // assume var count = list. Count; // arrange list. Add(item); // act Assert. Is. True(list. Count == count + 1); // assert }

Argument of MUT • Argument value of MUT may be used Pattern 2. 3: Constructor Test [Pex. Method] void Constructor(int capacity) { var list = new Array. List(capacity); // create Assert. Invariant(list); // assert invariant Assert. Are. Equal(capacity, list. Capacity); // assert }

Reciever or Argument of Earlier Method • Receiver or argument value of a method before invoking MUT value s parsed Pattern 2. 4/5: Roundtrip [Pex. Method] void To. String. Parse. Roundtrip(int value) { // two-way roundtrip string s = value. To. String(); int parsed = int. Parse(s); // assert Assert. Are. Equal(value, parsed); }

Observer Methods • Invoking observer methods on the modified object state Pattern 2. 6: State Relation [Pex. Method] void Insert. Contains(string value) { var list = new List<string>(); list. Add(value); Assert. Is. True(list. Contains(value)); } Each modified object property should be read by at least one observer method.

Observer Methods cont. • Forcing observer methods to return specific values (e. g. , true or false) can force you to add specific assumptions or scenarios [Pex. Method] void Push. Is. Full([Pex. Assume. Under. Test]UInt. Stack s, int value) { Pex. Assume. Is. True(s. Get. Size() == (s. Get. Max. Size()-1)); s. Push (value); Assert. Is. True(s. Is. Full ()); }

Alternative Computation • Invoking another method/method sequence to produce a value to be used Pattern 2. 7: Commutative Diagram [Pex. Method] void Commutative. Diagram 1(int x, int y) { // compute result in one way string z 1 = Multiply(x, y). To. String(); // compute result in another way string z 2 = Multiply(x. To. String(), y. To. String()); // assert equality if we get here Pex. Assert. Are. Equal(z 1, z 2); }

Divide and Conquer • Split possible outcomes into cases (each with pre and post condition) Pattern 2. 8: Cases [Pex. Method] void Business. Rules(int age, Job job) { var salary = Salary. Manager. Compute. Salary(age, job); Pex. Assert. Case(age < 30). Implies(() => salary < 10000). Case(job == Job. Manager && age > 35). Implies(() => salary > 10000). Case(job == Job. Manager && age < 20). Implies(() => false); }

Class Invariant Checker • If class invariant checker (rep. Ok) exists or you would be willing to write one, use it to assert Pattern 2. 3: Constructor Test [Pex. Method] void Constructor(int capacity) { var list = new Array. List(capacity); // create Assert. Invariant(list); // assert invariant Assert. Are. Equal(capacity, list. Capacity); // assert }

Other Patterns Pattern 2. 9: Allowed exceptions • • • [Pex. Allowed. Exception(typeof(Argument. Null. Exception))] [Expected. Exception(typeof(Argument. Null. Exception))] Pattern 2. 10: Reachability • • • [Pex. Expected. Goals] + throw new Pex. Goal. Exception(); Pattern 2. 11: Parameterized Stub No scenarios or assertions Pattern 2. 12: Input Output Test void Add(int a, int b, out int result) { … } int Substract(int a, int b) { … } Pattern 2. 13/14: Regression Tests bool Parse(string input) { … } Pex. Store. Value. For. Validation("result", result); http: //research. microsoft. com/en-us/projects/pex/patterns. pdf

69

Test-Driven Development (TDD) �Basic Idea: Write tests before code Refine code with new tests �In more detail, TDD is a cycle of steps: Add a test, Run it and watch it fail, Change the code as little as possible such that the test should pass, Run the test again and see it succeed, Refactor the code if needed.

Note: TDD and specifications �TDD encourages writing specifications before code Exemplary specification �Later, we will generalize TDD to Parameterized TDD Axiomatic specifications

Parameterized Test. Driven Development Write/refine Contract as PUT Bug in PUT Write/refine Code of Implementation Bug in Code Run Pex no failures Use Generated Tests for Regression failures Fix-it (with Pex), Debug with generated tests

Coding Duels 1, 767, 012 clicked 'Ask Pex!'

Coding Duels s t e r ec Pex computes “semantic diff” in cloud secret reference implementation vs. code written in browser You win when Pex finds no differences For more info, see our ICSE 2013 SEE paper: http: //taoxie. cs. illinois. edu/publications/icse 13 see-pex 4 fun. pdf

Behind the Scene of Pex for Fun behavior Secret Impl == Player Impl Secret Implementation class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } } 1, 594, 092 class Test { public static void Driver(int x) { if (Secret. Puzzle(x) != Player. Puzzle(x)) throw new Exception(“Mismatch”); } } Player Implementation class Player { public static int Puzzle(int x) { return x; } } 75

Code Hunt Programming Game https: //www. codehunt. com/

Code Hunt Programming Game

Code Hunt Programming Game

Code Hunt Programming Game

Code Hunt Programming Game

Code Hunt Programming Game

Code Hunt Programming Game

Code Hunt Programming Game

Code Hunt Programming Game

Code Hunt Programming Game

Code Hunt Programming Game

Code Hunt Programming Game

Code Hunt Programming Game

It’s a game! iterative gameplay adaptive personalized no cheating clear winning criterion code t e r sec test cases

Thank you! Questions ? taoxie@illinois. edu Materials: http: //taoxie. cs. illinois. edu/courses/testing/ http: //research. microsoft. com/pex