32 Automated Unit Test Generation for Classes with

/ 32 Automated Unit Test Generation for Classes with Environment Dependencies Andrea Arcuri, Gordon Fraser, Juan Pablo Galeotti ASE 2014 Presented by Yongbae Park

1 / 32 Overview • Motivation: low coverage & non-deterministic test results in automated test generation for a class interacting with environment 1. Branches depending on the environment that may not be covered by a sequence of method calls of a class under test 2. Passing test cases generated by a tool can fail as the environment changes • The environment is inputs from outside of a class that a class cannot control such as network, file system, system time, and user inputs • Approach: use a mock library interacting with the virtual environment to increase coverage and produce deterministic test result • Users of a test generation tool do not have to build their own mock library • The mock library provides deterministic inputs from the virtual environment separated from the real environment • Evaluation: the mock library increases branch coverage and decreases the number of test cases making non-deterministic results • Evo. Suite with the mock library produces test cases whose branch coverage is increased by 183% and the number of tests with non-deterministic test results is reduced by 99. 2%

2 / 32 Contents • Background • How Evo. Suite generates test suites using a genetic algorithm • Motivation • The limitation of Evo. Suite in a class with environment dependency • Approach • Mocking Java standard API to control the environment • Empirical study • Related Works • Conclusion

3 / 32 Background – Evo. Suite • Given class under test (CUT), Evo. Suite automatically generates a test suite using a genetic algorithm 1. 2. 3. 4. 5. 6. 7. 8. Create 2 initial test suites by adding method calls randomly and insert the test suites into current generation Select two test suites from current generation Create 2 new test suites by crossover (exchange test cases of the suites) Modify two test suites (test drivers) from step 3 with mutation operators (remove, insert, change operators) Insert the new two test suites from step 4 to next generation if coverage of the new two test suites are higher than that of parents Repeat 2~5 until next generation has sufficient number of test suites Repeat 2~6 until time limit is reached or all branches are covered Select a test suit with the highest branch coverage and insert assertions by observing differences in execution traces between the original CUT and a mutated CUT

4 / 32 Example for Evo. Suite • Message class contains 2 members: created meaning message creation date and text meaning contents of a message • The message creation date is set in the constructor of Message 1 public class Message { private Date created; 2 private String text; 3 public Message(String text) { 4 this. created=new Date(); 5 this. text=(text==null? ””: text); 6 } 7 public String to. String() { 8 return created+”, ”+text; 9 } 10 public String get. Text() { 11 return text; 12 13 } }

5 / 32 Initial Test Suite Generation • Evo. Suite randomly inserts a new statement in empty test cases of two initial test suites until the length of test cases reaches the maximum length • Evo. Suite add a method call whose callee is a method of a class under test or a method call of an object that is available at the end • A parameter of the created method call is selected from available values, null, or a random value • The maximum length for each test case is set by random number 1 public class Message { private Date created; 2 private String text; 3 public Message(String text) { 4 this. created=new Date(); 5 this. text=(text==null? ””: text); 6 } 7 public String to. String() { 8 return created+”, ”+text; 9 } 10 public String get. Text() { 11 return text; 12 13 } } public class Test. Suite 1 { public void test 0() { //length 3 Message v 0 = new Message(“e”); Message v 1 = new Message(“c”); String v 2 = v 1. to. String(); } public void test 1() { //length 2 Message v 0 = new Message(null); String v 1 = v 0. get. Text(); } }

6 / 32 Crossover • public class Test. Suite 1 { public void test 0() { Message v 0 = new Message(“e”); Message v 1 = new Message(“c”); String v 2 = v 1. to. String(); } public void test 1() { Message v 0 = new Message(null); String v 1 = v 0. to. String(); } } public class Test. Suite 2 { public void test 0() { Message v 0 = new Message(“a”); String v 1 = v 0. to. String(); String v 2 = v 1. trim(); } public void test 1() { Message v 0 = new Message(null); String v 1 = v 0. get. Text(); String v 2 = v 0. to. String(); } } public class Test. Suite 3 { public void test 0() { Message v 0 = new Message(“e”); Message v 1 = new Message(“c”); String v 2 = v 1. to. String(); } public void test 1() { Message v 0 = new Message(“a”); String v 1 = v 0. to. String(); String v 2 = v 1. trim(); } } public class Test. Suite 4 { public void test 0() { Message v 0 = new Message(null); String v 1 = v 0. to. String(); } public void test 1() { Message v 0 = new Message(null); String v 1 = v 0. get. Text(); String v 2 = v 0. to. String(); } }

7 / 32 Mutation – Insert Operator • Insert operator adds a new statement at a random position by using one of the two ways 1. Add a method call statement whose callee is a method of a class under test 2. Add a method call of an object that is available at the random position • A parameter of the created method call is selected from available values, null, or a random value public class Test. Suite 4 { public void test 0() { Message v 0 = new Message(null); String v 1 = v 0. to. String(); Adding a method call of } } a class under test public class Test. Suite 4 { public void test 0() { Message v 0 = new Message(null); String v 1 = v 0. to. String(); Message v 2 = new Message(“b”); } } Adding a method call of an available object public class Test. Suite 4 { public void test 0() { Message v 0 = new Message(“a”); String v 1 = v 0. to. String(); boolean v 2 = v 1. equals(null); } }

8 / 32 Mutation – Remove & Change Operator • Remove operator randomly selects a statement in a test case and remove the statement • Change operator randomly changes a callee or a parameter of a method invocation statement in a test case • Change operator selects a new callee method that whose return type is same as the original method • Change operator changes a parameter of a method call into a value which is available from the previous statements or a random value public class Test. Suite 4 { public void test 0() { Message v 0 = new Message(null); String v 1 = v 0. to. String(); } } callee method change public class Test. Suite 4 { public void test 0() { Message v 0 = new Message(null); String v 1 = v 0. get. Text(); } } parameter change (random value) public class Test. Suite 4 { public void test 0() { Message v 0 = new Message(“a”); String v 1 = v 0. to. String(); } }

9 / 32 Inserting Assertions • Evo. Suite adds assertions by observing differences between an execution trace of the original class under test and that of a mutated version of a class under test • An assertion is added if values of a variable (primitive or String type) are different in the two execution traces • Evo. Suite mutates the original class under test using the insert, remove, change operators in test generation

10 / 32 Inserting Assertions - Example public class Test. Suite 1 { public void test 0() { Message v 0 = new Message(null); String v 1 = v 0. to. String(); } } 1 public class Message { //original private Date created; 2 private String text; 3 public Message(String text) { 4 this. created=new Date(); 5 this. text=(text==null? ””: text); 6 } 7 public String to. String() { 8 return created+”, ”+text; 9 10 }…} Execution result of Message v 1 = “ 2014. 10. 02, ” 1 public class Message { //mutated private Date created; 2 private String text; 3 public Message(String text) { 4 this. created=new Date(); 5 6 7 } public String to. String() { 8 return created+”, ”+text; 9 10 }…} Execution result of mutated Message v 1 = “ 2014. 10. 02, null” assert. Equals(v 1, ” 2014. 10. 02, ”)

11 / 32 Generated Test Case • Evo. Suite creates a test case with an assertion containing a string of the created date of the test case • test 0() uses a string “ 2014. 10. 2, ” • However, the test case will fail when the test case is executed on a different day 1 public class Message { private Date created; 2 private String text; 3 public Message(String text) { 4 this. created=new Date(); 5 this. text=(text==null? ””: text); 6 } 7 public String to. String() { 8 return created+”, ”+text; 9 10 } } public class Test. Suite 1 { public void test 0() { Message v 0 = new Message(null); String v 1 = v 0. to. String(); assert. Equals(v 1, ” 2014. 10. 02, ”); } }

12 / 32 Motivation • The test result of a class interacting with the environment is non -deterministic (not reproducible) • The author defines the environment as inputs from outside of a class • E. g. a state of virtual machine (free & total memory space), a state of operating system (system time, file system, user inputs) • A test case is unstable if the test case passed when test case is generated but may fail in the next execution with different environment Operating System Virtual Machine Class under test Memory state Time & Date File system User inputs

13 / 32 Environment Mocking • “Mocking” is the replacement real classes with modified classes that behave deterministically • In the following example, Date is replaced by Mock. Date that always returns a fixed value 1 2 3 4 5 6 7 8 9 10 11 12 13 14 public class Message { private Date created; private String text; public Message(String text) { this. created=new Mock. Date(); this. text=(text==null? ””: contents); } public String to. String() { return created+”, ”+text; } } public class Mock. Date() { public String to. String() { return “ 2014. 10. 02”; } } public class Test. Suite 1 { public void test 0() { Message v 0 = new Message(null); String v 1 = v 0. to. String(); assert. Equals(v 1, ” 2014. 10. 02, ”); } }

14 / 32 Overview of Environment Mocking • A test generation tool with a generic mock library create a stable test suite with higher branch coverage without user’s (tester) efforts • A mock library of the paper mocks console inputs, file I/O, general API class of Java standard API • The mock library consist of a custom Input. Stream class for console inputs, 11 classes of file I/O (e. g. File), 12 general API classes (e. g. System, Runtime) • The mock library have helper methods that set the environment in a test case • The technique replaces standard library with the mock library using Java bytecode instrumentation • Insert operator of Evo. Suite inserts the helper methods in test cases to generate test suites that controls the initial environment

15 / 32 Console Inputs • A customized Input. Stream called System. In. Util has a helper method add. Input. Line(String) so that a test case can write contents of console inputs • The console contents of System. In. Util is reset before every test execution • In instrumentation, System. io in a class under test is changed into System. In. Util. io

16 / 32 File I/O • The authors mocked 11 JVM file I/O API classes • Mock classes are children classes of the mocked classes • The authors overridden the methods of the mocked classes to access virtual file system instead of real file system of operating system • For example, the authors overridden 37 methods among 52 methods of File class • Test cases become independent from each other and there is no negative side-effect such as file system corruption • The authors created helper methods such as append. Line. To. File() to control the initial state of the virtual file system java. io. File java. io. Print. Stream java. io. File. Input. Stream java. io. Print. Writer java. io. File. Output. Stream java. util. logging. File. Handler java. io. Random. Access. File javax. swing. JFile. Chooser java. io. File. Reader java. io. File. Writer javax. swing. filechooser. File. System. View

17 / 32 General JVM Calls • The authors mocked 12 JVM general API classes to control the environment such as time and random number Class name Environment java. lang. Exception java. lang. Throwable java. util. logging. Log. Record Stack trace message java. lang. Thread java. lang. Runtime Memory usage & the number of processors java. lang. System java. util. Date java. util. Calendar Current system time & date java. util. Gregorian. Calendar java. lang. Class java. lang. Math java. util. Random Reflection (the order of Method objects) Random number

18 / 32 Stack Trace (1/2) • Stack. Test has a method get. Stack. Trace() which returns a stack trace string of Throwable object • The message contains the name of a thread and the name of class that executes the test case • The test case fails if the name of a thread or test runner is changed 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 public class Stack. Test { public static String get. Stack. Trace(Throwable t) { Pure. String. Writer sw = new Pure. String. Writer(); Print. Writer pw = new Print. Writer(); t. print. Stack. Trace(pw); return sw. to. String(); } } public class Test. Suite { public void test 0() { String result = Stack. Test. get. Stack. Trace(new Throwable()); assert. Equal("Exception in thread “Thread-0" java. lang. Throwablen tat Test. Runner. run(Test. Runner. java: 120)n tat Test. Suite. test 0(Test. java: 10)" , result); } }

19 / 32 Stack Trace (2/2) • Stack trace message depends on the string representation of Thread object and JUnit runner • Stack trace message contains the string representation with thread id depending on the number of created thread • JUnit runner is a class that executes each test case and bottom of stack trace message contains the class name of JUnit runner class • A tester can use custom JUnit runner to modify test execution process • Mocked Thread, Exception, Throwable, and Log. Record create stack trace message deterministically • The mock class of Thread print the string representation deterministically • The mock class of Thread, Exception, Throwable, and Log. Record does not prints custom JUnit runner class in stack trace

20 / 32 Memory & Processor (1/2) • Performance executes a certain task and measure execution time and memory usage • get. Memory. Usage() returns a string containing the amount of memory used by the task • The test case fails if free memory space is changed by execution of a test runner or other test cases 1 2 3 4 5 6 7 8 9 10 public class Performance { public static String get. Memory. Usage() { perform. Task(); return "Memory: "+(Runtime. total. Memory()-Runtime. free. Memory()); } } public class Test. Suite { public void test 0() { String result = Performance. get. Memory. Usage(); assert. Equal(“Memory: 341”, result); } }

21 / 32 Memory & Processor (2/2) • Runtime provides system specific values such as the number of processors and memory usage • available. Processors() returns the number of available processors • free. Memory(), total. Memory(), and max. Memory() return memory usage • available. Processors(), free. Memory(), total. Memory(), and max. Memory() of a mock class of Runtime return constant values

22 / 32 System Time (1/2) • Message object contains the creation date of a message • The generated test case contains a string of the day when the test case was created • The test case fails if the test case is executed on 7 next day 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 public class Message { private Date created; private String msg; public Message(String msg) { created = new Date(); this. msg = msg; } public Date get. Created() { return created; } } public class Test. Suite { public void test 0() { Message msg = new Message(); assert. Equal(“ 2014. 10. 02”, msg. get. Created(). to. String()); } }

23 / 32 System Time (2/2) • Mock methods and classes of current. Time. Millis(), nano. Time(), Date, Calendar, and Geregorian. Calendar returns default time • The mock library has helper methods to set initial system time • Each call of the system time methods increases the time by 1

24 / 32 Reflection (1/2) • get. Available. Methods() returns a string of a list of public methods • If test 0() is executed before test 1(), the execution result of test 0() is pass • Otherwise, the execution result of test 0() is fail 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 public class Load. Library { public static String get. Available. Methods(Class c) { String result = “”; for (Method each : c. get. Methods() ) { result += each. get. Name() + “n”; } return result; } } public class Test. Suite { public void test 0() { String methods = get. Available. Methods(Test. Suite. class); assert. Equal(“test 0ntest 1”, methods); public void test 1() { … } }

25 / 32 Reflection (2/2) • The order of an array returned by reflection methods is non-deterministic • The order of an array that is returned by reflection methods (get. Classes(), get. Methods()) depends on method invocation order or class loader implementation • The reflection methods of a mock class of Class return sorted lists of available methods, classes, annotations, etc.

26 / 32 Random Number • random() returns position data in 2 D space randomly • The content of the returned object can be different from the previous execution • Mock classes of Math and Random return the next random number deterministically 1 2 3 4 5 6 7 8 9 10 11 12 13 public class Position { public int x, y; public static Position random() { Position result = new Position(); result. x = (int)Math. random(); result. y = (int)Math. random(); return result; } } public class Test. Suite { public void test 0() { Position p = Position. random(); assert. Equal(2, p. x); } }

27 / 32 Static Field Initialization • A class under test that modifies static field may change behavior of test cases if the execution order of the test cases is changed • Evo. Suite creates unstable test because Evo. Suite does not reset static field nor restart Java virtual machine • The extended Evo. Suite calls static initialization methods of classes whose static fields are modified to reset static field with low overhead • Every writing to a static field is instrumented with a method call that records the modification • A test suite generated by Evo. Suite calls the static initialization methods before the execution of each test case

28 / 32 Empirical Study • RQ 1: does controlling the environment successfully increase coverage on known cases of environmental interaction? • RQ 2: does controlling the environment resolve known issues of unstable tests? • RQ 3: How do results generalize to the SF 100 corpus? • The authors applied extended Evo. Suite to SF 100 corpus • SF 100 corpus is 100 Java projects randomly selected from Source. Forge • SF 100 corpus contains 11, 219 Java classes • Evo. Suite created test cases up to 3 minutes for each class

29 / 32 RQ 1: Coverage – Setup • The authors manually selected 30 classes from SF 100 corpus that interact with the environment • The classes that uses environment-related API are selected • The authors compared coverage of 6 different configurations • Base: the default Evo. Suite • Console: Evo. Suite with console input mocking • VFS: Evo. Suite with virtual file system • JVM: Evo. Suite with general API call mocking • Static: Evo. Suite with static field initialization • All: Evo. Suite with console input, virtual file system, general API call mocking and static field initialization

30 / 32 RQ 1: Coverage – Experiment Results • No. Base Extended Console VFS JVM Static All 1 0. 10 0. 76 0. 10 0. 78 16 0. 22 0. 74 0. 22 0. 73 2 0. 75 1. 00 17 0. 00 0. 87 3 0. 09 0. 08 0. 53 0. 08 0. 41 0. 80 18 0. 00 1. 00 4 0. 26 0. 74 0. 27 0. 25 0. 26 0. 73 19 0. 30 0. 33 0. 77 0. 30 0. 69 0. 79 5 0. 40 0. 89 0. 40 0. 99 20 0. 00 1. 00 6 0. 60 0. 61 0. 99 21 0. 79 0. 77 0. 67 0. 88 0. 85 7 0. 20 0. 77 22 0. 67 0. 68 0. 69 0. 61 0. 99 0. 96 8 0. 29 0. 40 0. 28 0. 29 0. 40 23 0. 08 0. 99 9 0. 94 0. 95 1. 00 24 0. 25 0. 87 0. 25 0. 99 10 0. 83 0. 95 0. 83 0. 94 25 0. 07 0. 56 11 0. 12 0. 70 26 0. 22 0. 66 0. 22 0. 63 12 0. 08 0. 68 0. 09 0. 11 0. 64 27 0. 28 0. 62 0. 30 0. 27 0. 25 0. 64 13 0. 24 0. 79 0. 86 28 0. 03 0. 56 0. 03 0. 54 14 0. 12 0. 93 29 0. 34 0. 79 0. 34 0. 80 15 0. 33 0. 80 30 0. 20 0. 79 Avg. 0. 29 0. 35 0. 65 0. 29 0. 37 0. 82

31 / 32 RQ 2: Unstable Tests – Setup • The authors manually selected 30 classes from SF 100 corpus from which the default Evo. Suite creates unstable tests • There are 2 classes that are used in both RQ 1 and RQ 2 • The authors compared coverage of 6 different configurations same as the experiments in RQ 1

32 / 32 RQ 2: Unstable Tests – Experiment Results • No. Base Extended Console VFS JVM Static All 1 2. 72 2. 52 2. 49 0. 00 16 1. 75 1. 73 1. 47 1. 57 0. 00 2 1. 14 1. 16 1. 20 2. 06 0. 00 17 1. 19 1. 23 1. 03 0. 00 1. 17 0. 00 3 2. 99 3. 04 3. 17 3. 21 0. 14 0. 13 18 2. 59 2. 87 2. 55 0. 09 2. 27 0. 11 4 1. 27 1. 28 1. 35 1. 74 0. 00 19 2. 49 2. 39 2. 93 0. 02 1. 62 0. 00 5 2. 37 2. 36 2. 33 2. 19 0. 00 0. 07 20 1. 91 1. 90 1. 87 0. 00 1. 77 0. 00 6 4. 20 4. 19 4. 24 4. 15 1. 55 0. 01 21 3. 96 3. 87 2. 40 3. 81 0. 00 7 1. 33 1. 40 1. 39 0. 00 1. 40 0. 00 22 0. 93 0. 78 0. 00 0. 80 0. 03 8 0. 64 0. 75 0. 79 0. 82 0. 06 0. 08 23 1. 82 1. 59 1. 28 0. 00 1. 55 0. 00 9 0. 84 0. 98 0. 75 0. 00 0. 77 0. 00 24 3. 26 3. 67 3. 66 0. 00 3. 38 0. 00 10 7. 16 7. 09 6. 93 0. 03 6. 94 0. 00 25 1. 83 1. 76 1. 79 0. 12 1. 60 0. 16 11 0. 62 0. 61 0. 00 0. 62 0. 00 26 2. 13 2. 12 2. 21 0. 00 2. 19 0. 00 12 2. 50 2. 52 2. 38 0. 00 27 1. 91 1. 86 2. 18 1. 79 0. 01 0. 00 13 5. 57 5. 47 5. 87 0. 00 28 1. 63 2. 01 1. 87 0. 86 1. 22 0. 00 14 3. 73 3. 66 3. 41 0. 02 1. 73 0. 00 29 1. 22 0. 00 1. 19 0. 00 15 6. 86 6. 79 7. 50 4. 41 7. 05 0. 30 30 0. 57 0. 54 0. 57 0. 67 0. 00 Avg. 2. 43 2. 45 2. 40 1. 27 1. 30 0. 02

33 / 32 RQ 3: Generalization • The authors compared the default Evo. Suite (base) and the extended Evo. Suite (all) by applying both of them to all 11, 219 classes of SF 100 corpus • The branch coverage increased by 1. 8% (=(77. 9%-76. 5)/76. 5) • The default Evo. Suite achieved 76. 5% of branch coverage and the extended Evo. Suite achieved 77. 9% • The extended Evo. Suite achieved 2, 803 additional branches • The number of classes from which Evo. Suite creates unstable tests is reduced by 53. 8% (=(1, 452 -671)/1, 452) • The default Evo. Suite created unstable tests from 1, 452 classes while the extended Evo. Suite created unstable tests from 671 classes

34 / 32 Related Works • Mocking frameworks help developers write mock classes of user classes, not class of the standard library • The mocking frameworks such as Moles (C#), Jmock (Java) provide mechanisms to create mock class and replace the original classes with the mock classes • The developers should change mock classes as the original class changes • The paper aims mocking Java standard API which does not change frequently • Once mock library is built, the mock library can be applied to any class under test

35 / 32 Conclusion • Mocking library interacting with the virtual environment increases coverage and reduces unstable tests in automated unit test generation • The modified Evo. Suite creates test cases that controls initial environment to increase branch coverage • The generated test cases are stable because interactions with the virtual environment is deterministic • The authors shown the empirical results that using mocking library increase branch coverage and reduces unstable tests from Evo. Suite for 100 Java projects