Eclat Automatic Generation and Classification of Test Inputs
- Slides: 50
Eclat: Automatic Generation and Classification of Test Inputs Carlos Pacheco and Michael Ernst Program Analysis Group MIT
The Problem ● ● Suppose you have a program that works – It passes an existing test suite – Its observable behavior appears correct You want improved confidence in the program's reliability – Ensure that the program works on different inputs ● If program’s operation incorrect on some input, fix the program and add a new test case to test suite
Input Generation ● Can automatically generate test inputs – – ● Random generation [Klaessen & Hughes 2002, …] Bounded exhaustive testing [Boyapati et al. 2002] Constraint solving [Korel 1996, Gupta 1998, …] Many more… Without automatic tool support, must inspect each resulting input (unless executable spec/oracle exists) – – Is the input fault-revealing? Is the input useful?
Research Goal ● Help the user select from a large number of inputs, a small “promising” subset: – Inputs exhibiting new program behavior – Inputs likely to reveal faults
The Technique execution (e. g. test suite) Model generator model of correct operation potentially fault revealing inputs program candidate inputs Input generator Classifier illegal fault normal Reducer True faults False alarms
The Technique execution (e. g. test suite) Model generator model of correct operation potentially fault revealing inputs program candidate inputs Input generator Classifier illegal fault normal Reducer True faults False alarms
The Technique execution (e. g. test suite) Model generator model of correct operation potentially fault revealing inputs program candidate inputs Input generator Classifier illegal fault normal Reducer True faults False alarms
The Technique execution (e. g. test suite) Model generator model of correct operation potentially fault revealing inputs program candidate inputs Input generator Classifier illegal fault normal Reducer True faults False alarms
The Technique execution (e. g. test suite) Model generator model of correct operation potentially fault revealing inputs program candidate inputs Input generator Classifier illegal fault normal Reducer True faults False alarms
The Technique execution (e. g. test suite) Model generator model of correct operation potentially fault revealing inputs program candidate inputs Input generator Classifier illegal fault normal Reducer True faults False alarms
The Technique execution (e. g. test suite) Model generator model of correct operation potentially fault revealing inputs program candidate inputs Input generator Classifier illegal fault normal Reducer True faults False alarms
The Technique execution (e. g. test suite) Model generator model of correct operation potentially fault revealing inputs program candidate inputs Input generator Classifier illegal fault normal Reducer True faults False alarms
The Technique execution (e. g. test suite) Model generator model of correct operation potentially fault revealing inputs program candidate inputs Input generator Classifier illegal fault normal Reducer True faults False alarms
Model Generator ● Our technique uses a model generator to produce a model of correct program operation, derived from observing a correct execution [Ernst et al. 2001, Ammons et al. 2002, Hankel and Diwan 2003, …] ● Our technique requires – Set of properties hold at component boundaries – The properties can be evaluated
Example: bounded stack [Stotts et al. 2002, Xie and Notkin 2003, Csallner and Amaragdakis 2004] public class Stack { private int[] elems; private int top. Of. Stack; private int capacity; public Stack() {. . . } public void push(int k) {. . . } public void pop() { top. Of. Stack --; } public boolean is. Member(int i) {. . . } public class Stack. Test {. . . }
Example: bounded stack public class Stack { private int[] elems; private int top. Of. Stack; private int capacity; public Stack() {. . . } public void push(int k) {. . . } public void pop() { top. Of. Stack --; } public boolean is. Member(int i) {. . . } object properties capacity == elems. length elems != null capacity == 2 top. Of. Stack >= 0 pop: entry properties elems Î { [3, 0], [3, 2] } is. Member: entry properties k Î elems is. Member: exit properties elems == orig(elems) orig(k) Î elems public class Stack. Test {. . . }
Classifier execution (e. g. test suite) Model generator model of correct operation potentially fault revealing inputs program candidate inputs Input generator Classifier illegal fault normal Reducer True faults False alarms
Classifier ● Run program on candidate input ● Detect set of violated model properties ● Classify: entry exit Classification violations? no no normal no yes fault yes no normal (new) yes illegal
Classifier: normal input exit entry Classification violations? no no normal no yes fault yes no normal (new) yes illegal object properties (all methods) capacity == elems. length elems != null capacity == 2 top. Of. Stack >= 0 pop: entry properties elems Î { [3, 0], [3, 2] } Stack var 1 = new Stack(); var 1. push(3); var 1. pop(); is. Member: entry properties k Î elems is. Member: exit properties elems == orig(elems) orig(k) Î elems
Classifier: normal input exit entry Classification violations? no no normal no yes fault yes no normal (new) yes illegal object properties (all methods) capacity == elems. length elems != null capacity == 2 top. Of. Stack >= 0 pop: entry properties elems Î { [3, 0], [3, 2] } Stack var 1 = new Stack(); var 1. push(3); var 1. pop(); is. Member: entry properties k Î elems is. Member: exit properties elems == orig(elems) orig(k) Î elems
Classifier: fault-revealing input exit entry Classification violations? no no normal no yes fault yes no normal (new) yes illegal object properties (all methods) capacity == elems. length elems != null capacity == 2 top. Of. Stack >= 0 pop: entry properties elems Î { [3, 0], [3, 2] } Stack var 1 = new Stack(); var 1. push(3); var 1. pop(); is. Member: entry properties k Î elems is. Member: exit properties elems == orig(elems) orig(k) Î elems
Classifier: fault-revealing input exit entry Classification violations? no no normal no yes fault yes no normal (new) yes illegal object properties (all methods) capacity == elems. length elems != null capacity == 2 top. Of. Stack >= 0 (on exit) pop: entry properties elems Î { [3, 0], [3, 2] } Stack var 1 = new Stack(); var 1. push(3); var 1. pop(); is. Member: entry properties k Î elems is. Member: exit properties elems == orig(elems) orig(k) Î elems
Classifier: illegal input exit entry Classification violations? no no normal no yes fault yes no normal (new) yes illegal object properties (all methods) capacity == elems. length elems != null capacity == 2 top. Of. Stack >= 0 pop: entry properties elems Î { [3, 0], [3, 2] } Stack var 1 = new Stack(); var 1. push(0); var 1. is. Member(-5); is. Member: entry properties k Î elems is. Member: exit properties elems == orig(elems) orig(k) Î elems
Classifier: illegal input exit entry Classification violations? no no normal no yes fault yes no normal (new) yes illegal object properties (all methods) capacity == elems. length elems != null capacity == 2 top. Of. Stack >= 0 pop: entry properties elems Î { [3, 0], [3, 2] } Stack var 1 = new Stack(); var 1. push(0); var 1. is. Member(-5); is. Member: entry properties k Î elems is. Member: exit properties elems == orig(elems) orig(k) Î elems
Reducer execution (e. g. test suite) Model generator model of correct operation potentially fault revealing inputs program candidate inputs Input generator Classifier illegal fault normal Reducer True faults False alarms
Reducer ● ● ● Partitions inputs based on the set of properties they violate Reports one inputs from each partition Inputs in same partition are likely to manifest same faulty behavior
Reducer: example Two equivalent inputs: Stack var 1 = new Stack(); var 1. push(3); var 1. pop(); Stack var 1 = new Stack(); var 1. push(0); var 1. pop(); var 1. push(3); var 1. pop(); Violation pattern: top. Of. Stack >= 0 (on exit)
Input Generator execution (e. g. test suite) Model generator model of correct operation potentially fault revealing inputs program candidate inputs Input generator Classifier illegal fault normal Reducer True faults False alarms
Bottom-up Random Generator 1. pool : = a set of primitives (null, 0, 1, etc. ) 2. do N times: 2. 1. create new inputs by calling methods/constructors using pool inputs as arguments 2. 2. add resulting inputs to the pool Null, 0, 1, 2, 3
Bottom-up Random Generator 1. pool : = a set of primitives (null, 0, 1, etc. ) 2. do N times: 2. 1. create new inputs by calling methods/constructors using pool inputs as arguments 2. 2. add resulting inputs to the pool Null, 0, 1, 2 3 Stack var 1 = new Stack(); Stack var 2 = new Stack(3);
Bottom-up Random Generator 1. pool : = a set of primitives (null, 0, 1, etc. ) 2. do N times: 2. 1. create new inputs by calling methods/constructors using pool inputs as arguments 2. 2. add resulting inputs to the pool Null, 0, 1, 2 3 Stack var 1 = new Stack(); var 1. is. Member(2); var 1. pop(); Stack var 2 = new Stack(3); var 2. push(3);
Bottom-up Random Generator 1. pool : = a set of primitives (null, 0, 1, etc. ) 2. do N times: 2. 1. create new inputs by calling methods/constructors using pool inputs as arguments 2. 2. add resulting inputs to the pool Null, 0, 1, 2 3 Stack var 1 = new Stack(); var 1. is. Member(2); var 1. pop(); var 1. equals(var 2); Stack var 2 = new Stack(3); var 2. push(3); var 2. pop(); var 2. push(0);
Avoiding illegal inputs ● It’s important that the inputs in the pool are legal – Inputs in the pool are building blocks for other inputs Input 1(tests pop) Stack s = new Stack(); s. pop(); fault-revealing Input 2 (tests equals) Stack s = new Stack(); s. pop(); Stack s 2 = new Stack(3); s. equals(s 2); illegal Input 3 (tests is. Member) Stack s = new Stack(); s. pop(); s. is. Member(1); illegal
execution (e. g. test suite) Using the classifier for generation Model generator model of correct operation potentially fault revealing inputs program candidate inputs Input generator Classifier illegal fault normal Reducer True faults False alarms
execution (e. g. test suite) Using the classifier for generation Model generator model of correct operation potentially fault revealing inputs program candidate inputs Input generator Classifier illegal fault normal Reducer True faults False alarms
Enhanced generator 1. pool : = a set of primitives (null, 0, 1, etc. ) 2. do N times: 2. 1. create new inputs by calling methods/constructors using pool inputs as arguments 2. 2 classify inputs 2. 3 throw away illegal inputs 2. 4 save away fault inputs 2. 5. add normal inputs to the pool
Eclat ● ● Java program Execution (e. g. test suite) Eclat Text output or XML output or JUnit tests Eclat generates inputs for Java unit testing Eclat uses the Daikon invariant detector to create a model of correct execution Each test input is wrapped as a JUnit test Eclat proposes assertion checks based on violated properties http: //pag. csail. mit. edu/eclat
Eclat’s output: example public void test_1_integrate() { Rat. Poly rp 1 = new Rat. Poly(4, 3); Rat. Poly rp 2 = new Rat. Poly(1, 1); Rat. Poly rp 3 = rp 1. add(rp 2); check. Pre. Properties(rp 3); rp 3. integrate(0); check. Post. Properties(rp 3); }
Eclat’s output: example public void test_1_integrate() { Rat. Poly rp 1 = new Rat. Poly(4, 3); Rat. Poly rp 2 = new Rat. Poly(1, 1); Rat. Poly rp 3 = rp 1. add(rp 2); check. Pre. Properties(rp 3); rp 3. integrate(0); check. Post. Properties(rp 3); } Assertion violation!
Eclat’s output: example public void test_1_integrate() { Rat. Poly rp 1 = new Rat. Poly(4, 3); Rat. Poly rp 2 = new Rat. Poly(1, 1); Rat. Poly rp 3 = rp 1. add(rp 2); check. Pre. Properties(rp 3); rp 3. integrate(0); check. Post. Properties(rp 3); Assertion violation! } public void check. Post. Properties(Rat. Poly rp) {. . . // on exit: all terms in rp always have non-zero coefficient Assert. assert. True(!all. Zeroes(rp. terms)); }
Eclat’s output: example public void test_1_integrate() { Rat. Poly rp 1 = new Rat. Poly(4, 3); Rat. Poly rp 2 = new Rat. Poly(1, 1); Rat. Poly rp 3 = rp 1. add(rp 2); check. Pre. Properties(rp 3); rp 3. integrate(0); check. Post. Properties(rp 3); Assertion violation! } public void check. Post. Properties(Rat. Poly rp) {. . . // on exit: all terms in rp always have non-zero coefficient Assert. assert. True(!all. Zeroes(rp. terms)); } 94 inputs violate this property. Of these, 3 are shown to the user.
Evaluation ● We used Eclat to generate inputs for – 6 families of libraries – 64 distinct interfaces – 631 implementations – 75, 000 NCNB lines of code
execution (e. g. test suite) Evaluation results: generator Model generator model of correct operation potentially fault revealing inputs program candidate inputs 1338 Input generator Classifier illegal fault normal Reducer True faults False alarms
execution (e. g. test suite) Evaluation results: classifier Model generator model of correct operation potentially fault revealing inputs program candidate inputs Classifier illegal fault normal Reducer 1338 Input generator illegal: 321 78% precision fault: 31 12% precision normal: 986 90% precision True faults False alarms
Evaluation results: reducer execution (e. g. test suite) Model generator model of correct operation potentially fault revealing inputs program candidate inputs Classifier illegal fault normal Reducer Input generator 30% precision illegal: 321 78% precision fault: 31 12% precision normal: 986 90% precision True faults False alarms 5 1338 1. 5 3. 5
Evaluation results: reducer execution (e. g. test suite) Model generator model of correct operation potentially fault revealing inputs program candidate inputs Classifier illegal fault normal Reducer Input generator 30% precision illegal: 321 78% precision fault: 31 12% precision normal: 986 90% precision True faults False alarms 5 1338 1. 5 3. 5
Future Directions ● Incorporate other classification techniques [Podgursky et al. 2003, Brun and Ernst 2004, Bowring et al. 2004, …] ● Incorporate other generation strategies [ Korel 1996, Gupta 1998, Klaessen & Hughes 2002, Boyapati et al. 2002, …]
Conclusion ● ● ● Technique to generate new program inputs – Inputs likely to reveal faults – Inputs not covered by an existing test suite Technique is effective in uncovering errors Eclat: automatically generates unit tests for Java classes Eclat: http: //pag. csail. mit. edu/eclat
Related Work ● ● ● Harder et al. Improving test suites via operational abstraction. ICSE 2003. Xie and Notkin. Tool-assisted unit test selection based on operational violations. ASE 2003. Csallner and Smaragdakis. JCrasher: an automatic robustness tester for Java. Software Practice and Experience, 2004.
- Toon calders
- Eclat algorithm
- Eclat bfc damparis
- What is agc in power system
- Automatic thesaurus generation
- First generation antipsychotics
- Lord your mercy endureth forever
- Feedback-directed random test generation
- Inputs and outputs of calvin cycle
- Input output design
- Inputs and outputs of aggregate planning
- Krebs cycle output
- Input and output of krebs cycle
- The human inputs and outputs information through
- In a banyan switch for 8 inputs and 8 outputs we have
- Direct and manage project work inputs
- Drainage basin inputs and outputs
- Inputs outputs transformation processes and feedback
- Improving productivity at starbucks
- Welfare inputs and outputs
- Aggregate planning inputs
- Fmea inputs and outputs
- Inputs and outputs
- Inputs and outputs
- When demand is unlimited and products use the same inputs
- What are the inputs to and outputs of a supply chain
- Water cycle
- Inputs of light reactions in photosynthesis
- Reactants in photosynthesis
- Mrp process
- Input fijo
- Sfas 142
- Inputs to develop project charter
- Production with two variable inputs
- Inputs, activities, outputs, outcomes
- Exe: automatically generating inputs of death
- Exe: automatically generating inputs of death
- Energy flow in cellular respiration
- Marginal rate of technical substitution formula
- Production with two variable inputs
- Kombinasi biaya terendah
- Splunk set diff
- Strategic management inputs
- What are the inputs into the evaluation phase of training?
- Coenzymes in cellular respiration
- Plan schedule management inputs
- Strategic management inputs
- Walmart cost leadership
- A homogenous
- Strategic inputs
- What are manual input devices