Java Specification Extension for Automated Test Development Igor

Java Specification Extension for Automated Test Development Igor B. Bourdonov, Alexei V. Demakov, Andrei A. Jarov, Alexander S. Kossatchev, Victor V. Kuliamin, Alexander K. Petrenko, Sergei V. Zelenov ISP RAS

J@va Design Goals n To preserve and to extend the techniques developed in previous verification projects n n Automatic generation of test oracles from pre- and postconditions Testing mechanism based of FSM traversal Test coverage metrics definition and tracking Dynamic test optimization for the target coverage n To propose a specification based test development process for use in the software industry n n To provide a specification extension of widely used programming language To simplify the technology Victor Kuliamin ISP RAS 2

Solutions Proposed n Constraint specifications and axioms are both used for functional requirements recording n n n Constraint specifications give rise to test oracles Axioms can be transformed into test scenarios OO architecture is used for specification reuse n User should specify branches as the basic test coverage metric based on specifications. Several additional metrics can be extracted automatically. n Test sequence can be generated with the help of FSM traversal mechanism. Test scenarios serve as a source for FSM description. n Flexible mechanism to bind specification with target component is used Victor Kuliamin ISP RAS 3

J@va Verification Suite Components Requirements recording Axioms Test effectiveness measure Constraint Specifications Test Scenarios Oracles Test drivers Legend Implementation FSM Traversal Algorithms Test sequence generation Handmade component Dependency Victor Kuliamin Test Coverage Metrics Test coverage trackers Generated component Prebuilt component Automatic derivation ISP RAS 4

Main Features of J@va n State based behavior specifications n Object state and invariants n Precondition and postcondition of a method n Axioms and algebraic specifications n Test coverage metrics description n Abstraction management n Test scenarios executed with the help of FSM based testing machine n Flexible OO test suite architecture Victor Kuliamin ISP RAS 5

Specification of Object State Attributes of class and invariants constitute the specification of state for objects of this class. Example. Specification of a queue with elements having priorities. Victor Kuliamin ISP RAS 6

Example of Object State Specification public class Priority. Queue. Specification { public List items; // The list of queue elements public List priorities; // The list of elements’ priorities invariant I 1() // The lists of elements and priorities have the same sizes { return items. size() == priorities. size(); } invariant I 2() // The list of priorities contains only Priority objects { for(int i = 0; i < priorities. size(); i++) if(!(priorities. element. At(i) instanceof Priority)) return false; return true; } invariant I 3() // The values of priorities should not increase { for(int i = 1; i < priorities. size(); i++) if( ((Priority)priorities. element. At(i)). greater( (Priority)priorities. element. At(i-1) ) ) return false; return true; } } Victor Kuliamin ISP RAS 7

Method Precondition and Postcondition Method precondition formulates the conditions when this method can be called. Method postcondition gives the criterion of the correct method’s behavior in terms of states of method parameters and other objects before the call and after it. Example. The specifications of enq() method in the queue. The most coarse test coverage is based on branch operators, which make a section of postcondition control flow graph. Victor Kuliamin ISP RAS 8

Example of Method Behavior Specification specification public synchronized void enq(Object obj) reads obj updates items. ? { pre { return obj != null; } // We cannot add null object reference to the queue post { branch “Single branch"; List new_items = (List)(@items. clone()); // @<expr> means pre-value of <expr> new_items. add. Last. Element(obj); return items. equals(new_items); } } Victor Kuliamin ISP RAS 9

Axioms and Algebraic Specifications Axioms are represented as special methods that return boolean result and can have precondition. Algebraic specifications are represented as special methods with several bodies. They also can have precondition. Example. Axiom and algebraic specification for a stack. Victor Kuliamin ISP RAS 10

Example of Axiom and Algebraic Specification axiom Push. And. Size() { pre { return true; } int old. Size = size(); push(new Object()); return size() == old. Size() + 1; } equivalence Object Push. And. Pop(Object obj) { pre { return true; } alternative { push(obj); return pop(); } alternative { return obj; } } Victor Kuliamin ISP RAS 11

Test Coverage Metrics Four predefined test coverage metrics are extracted from the control flow of pre- and postcondition. User can describe his own metrics in addition. Test execution environment keeps track of all the metrics defined and can perform dynamic optimizations for the coverage specified as target. Example. Specification of enq() operation for the queue with additional coverage metrics. Victor Kuliamin ISP RAS 12

Test Coverage Metrics Example specification public synchronized void enq(Object obj) reads obj updates items. ? { pre { return obj != null; } // We cannot add null object reference to the queue post { switch(items. size()) { case 0: mark “Empty queue”; break; case 1: mark “Single element in the queue”; break; default: mark “Several elements in the queue”; break; } branch “Single branch"; List new_items = (List)(@items. clone()); // @<expr> means pre-value of <expr> new_items. add. Last. Element(obj); return items. equals(new_items); } } Victor Kuliamin ISP RAS 13

Abstraction Management J@va has the following instruments for abstraction management n Inheritance of specifications n Refinement of specifications n Multi-level system of specifications As an example of inheritance we can present specifications of the ordinary queue and the queue with priorities, which has an additional operation enq(Object o) adding an element with minimum priority. Victor Kuliamin ISP RAS 14

Test Scenarios Test scenarios provide a powerful tool of test design. They are used as a source for test sequence generation. Test scenarios are executed with the help of the FSM traversal mechanism. Example. Test scenario for the queue. It is based of the FSM model of the queue, where different states correspond to different numbers of elements. The test generated from this scenario performs the traversal of the FSM having 10 states and enq() and deq() transitions from every state, except for the initial and last ones. The initial state has only enq() transition, and the last state has only deq() transition. Victor Kuliamin ISP RAS 15

Example of Test Scenario scenario public class Queue. Test. Scenario { public Queue. Specification queue = new Queue. Specification(); // The object under test public max. Number. Of. Items = 10; // The maximum number of elements in // the queue during the test public synchronized Abstract. Gen. State get. Gen. State() // Returns the state object { return new Int. Gen. State(queue. items. size()); // Library class Int. Gen. State is used } scenario public boolean enq() // Subscenario for enq() method { if( queue. items. size() < max. Number. Of. Items ) queue. enq(new Object()); return true; } scenario public boolean deq() { if( queue. items. size() != 0 ) queue. deq(); // Subscenario for deq() method return true; } } Victor Kuliamin ISP RAS 16

J@va Test Suite Architecture Specification Model Oracle Test coverage trackers Test Implementation Mediator Java Mediator Test Scenario Test driver Legend Handmade component Reference Victor Kuliamin Test Engine Generated component Inheritance ISP RAS Prebuilt component Automatic derivation 17

J@T — J@va Based Test Development Framework Victor Kuliamin ISP RAS 18

References 1. 2. 3. 4. 5. 6. A. K. Petrenko, I. B. Bourdonov, A. S. Kossatchev, V. V. Kuliamin. Experiences in using testing tools and technology in real-life applications. Proceedings of SETT’ 01, India, Pune, 2001 I. B. Bourdonov, A. S. Kossatchev, V. V. Kuliamin. Using Finite State Machines in Program Testing. "Programmirovanije", 2000, No. 2 (in Russian). Programming and Computer Software, Vol. 26, No. 2, 2000, pp. 61 -73 (English version) I. Bourdonov, A. Kossatchev, A. Petrenko, and D. Galter. KVEST: Automated Generation of Test Suites from Formal Specifications. Proceedings of World Congress of Formal Methods, Toulouse, France, LNCS, No. 1708, 1999, pp. 608 -621 I. B. Bourdonov, A. S. Kossatchev, V. V. Kuliamin, A. V. Maximov. Testing Programs Modeled by Nondeterministic Finite State Machine. (see [6] white papers) http: //www. fmeurope. org/databases/fmadb 088. html http: //www. ispras. ru/~Red. Verst/ Victor Kuliamin ISP RAS 19

Contacts Victor V. Kuliamin E-mail: kuliamin@ispras. ru 109004, B. Kommunisticheskaya, 25 Moscow, Russia. Web: http: //ispras. ru/~Red. Verst Phone: 007 (095) 912 -5317 ext 4422 Fax: 007 (095) 912 -1524 Victor Kuliamin ISP RAS 20