Random Test Generation of Unit Tests Randoop Experience

Random Test Generation of Unit Tests: Randoop Experience Wujie Zheng wjzheng@cse. cuhk. edu. hk 1

Outline n n Automated Test Generation of Unit Tests Randoop: Feedback-directed Random Test Generation Randoop Experience Potential Improvements 2

Outline n n Automated Test Generation of Unit Tests Randoop: Feedback-directed Random Test Generation Randoop Experience Potential Improvements 3

Automated Test Generation of Unit Tests n Problem Given a method of a class, how to automatically generate test cases to execute most of its code, i. e. , to achieve high code coverage. 4

Automated Test Generation of Unit Tests n A simple example class under test: java. util. Stack method under test: public synchronized int search(Object o) { int i = last. Index. Of(o); if (i >= 0) { return size() - i; } return -1; } 5

Automated Test Generation of Unit Tests n What need to be generated? q Test Inputs n q not only primitive values but also objects, including the receiver object Test Oracles n n Could use some basic contracts For not much test inputs, manually added by testers 6

Automated Test Generation of Unit Tests n An example test case // a test case for Stack. search(Object) // existing (previous generated) sequences that create parameters Stack var 0 = new Stack(); String var 1 = "hi!"; var 0. push((Object)var 1); // the method under test int var 2 = var 0. search(var 1); // test oracles assert. True(var 0. equals(var 0)); assert. True(var 2==1); // Regression assertion 7

Automated Test Generation of Unit Tests n Main solutions q Random Approaches n Randomly create and mutate objects q n q For a method under test, select objects created in existing sequences as its parameters Simple, scalable, but may not efficient Systematic Approaches n n Explore the possible paths systematically, for each path, collect the path condition and solve it to generate test inputs Potential high coverage, but complex 8

Outline n n Automated Test Generation of Unit Tests Randoop: Feedback-directed Random Test Generation Randoop Experience Potential Improvements 9

Randoop: Feedback-directed Random Test Generation n The basic idea q q Undirected random test generation is not efficient because it does not prune redundant or error objects (generated by duplicate or different method sequences) Identify redundant or error objects early can prune large search space 10

Randoop: Feedback-directed Random Test Generation n The basic idea q Identify redundant objects n q Use the equals() method Identify error objects n Exceptions. q n Null q q Exceptions frequently correspond to precondition violations for a method, and therefore there is little point extending them Null dereference exceptions caused by using null as an argument are often uninteresting Set the redundant or error objects as not extensible 11

Randoop: Feedback-directed Random Test Generation n Sequence generation for a method q q To execute a method m(T 1, …, Tk), we need to find existing sequences that generate the parameters For each Ti n n q Primitive type, randomly select a value from a pool Reference type, select a extensible (non-redundant and legal) value v from existing sequences, or use null Generate a new sequence, including the method m and the required sequences to generate the parameters 12

Randoop: Feedback-directed Random Test Generation n The Randoop algorithm Repeat until timeout q Randomly select a method m(T 1, …, Tk) q Generate a new sequence for m q Execute the new sequence q Check simple oracles, set the extensible flags of the objects Output the generated sequences as unit tests 13

Outline n n Automated Test Generation of Unit Tests Randoop: Feedback-directed Random Test Generation Randoop Experience Potential Improvements 14

Randoop Experience n Subject programs q q n Java. util Berkeley DB Interested Metrics q q q Number of tests Code coverage Error detection 15

Randoop Experience n Experiments on Java. util q q q n 7 classes in Java. util, used in other researchers’ experiments Run Randoop 60 seconds 4834 tests 50. 8% instruction coverage 195 failed tests, many false positives! Demo 16

Randoop Experience n Experiments on Berkeley DB q q q q Java edition of Berkeley DB, com. sleepycat. je Run Randoop 60 seconds Around 20, 000 tests 30% instruction coverage 300 failed tests, many false positives! Need to prepare some environments, e. g. , files in the disk Can more tests achieve higher coverage? n n Run Randoop 120 seconds: out of memory Demo 17

Outline n n Automated Test Generation of Unit Tests Randoop: Feedback-directed Random Test Generation Randoop Experience Potential Improvements 18

Potential Improvements n The problems of Randoop q Not enough code coverage n q Difficult to generate interesting sequences quickly before running out of memory Too large test suites n Need to perform test selection (reduction) 19

Thank you! 20