SATEL A Test Intention Language for ObjectOriented Specifications

SATEL A Test Intention Language for Object-Oriented Specifications of Reactive Systems Levi Lúcio Thesis’ defense Supervisor: Prof. Didier Buchs 1

Presentation Structure • • • Introduction State of the Art Problem Statement SATEL Discussion 2

Introduction Software Construction • Software Engineering is a discipline under intensive development • Large software projects present important challenges: – – – Operating systems Banking systems CERN detectors Automotive / Aviation Games Anatomy of a large project • • 1 M – 40 M € 20 – 100 engineers 1 – 5 years 5 – 10+ languages Multiple teams Multiple sites Complex communication 3

Introduction The spiral model (Boehm) 1. Understand the requirements 2. Build the Model 4. Verify (e. g. test) 3. Build the implementation 4

Introduction Catastrophic “bugs” In 1996 the Ariane 5 rocket explodes 37 s after takeoff: – Numerical data conversion error (insufficient integration testing) – € 500 M in hardware + € 7 B in development 5

Introduction Why do “bugs” exist? • The limits of Software Engineering are rapidly reached when building complex systems “The mythical man-month”, Brooks, 1975 • Building software is (still) a subjective activity: – – – Interpretation of the problem Conception of the architecture of the system Choice of the software and hardware tools Coding Quality measurement 6

Introduction Our Vision • • Scientifically developed tools are needed Strong formal theories induce better tools Testing is partially automatable Testing should be formal, too (inspired Gaudel) – The verification problem is undecidable – Heuristics (hypothesis) are needed to test – Qualification / Quantification of heuristics allows measuring quality 7

Presentation Structure • • • Introduction State of the Art Problem Statement SATEL Discussion 8

State of the Art Model Based Testing (MBT) Artifacts • Model • SUT (observed) • Implementation relation • Test derivation function • Test cases (Set) • Oracle • Test Purposes 9

State of the art Real / Formal Worlds Some minimal hypothesis are necessary to connect the real and the formal worlds! 10

State of the Art MBT theories BGM ioco Conformance Testing Coopn. Test Models Algebraic Specifications LTS (Labeled Transition Systems) CO-OPN/2 SUTs Pairs of equalities/inequalities between observable implemented sorts IOTS (input Output Transition Systems) LTS Impl. Relation Notion of -Algebra ioco (input output conformance) Bisimulation Test derivation Function Generate any pairs of terms of the satisfying the spec’s axioms ioco test derivation algorithm Generate all HML formula x {true, false} using the spec’s signature and semantics Test cases Pairs of terms LTS with leaves including the test verdict {pass, fail} HML formula x {true, false} Oracle (satisfiability) Each test case corresponds to an equality in the SUT For each test case synchronous execution with SUT reaches all pass leaves true (resp. false) HML formulas are satisfied (resp. not satisfied) by the SUT Test Purposes Uniformity/Regularity hypothesis on the axiom’s variables Incomplete test cases Uniformity/Regularity hypothesis on the HML formula’s variables Minimal hypothesis Implementation is a finitely gen. -Algebra Implementation can be modeled by an IOTS Implementation can be modeled 11 by a LTS

State of the Art BGM Exhaustive Test Set Reduction (P╞ SP) <=> (P╞o TSP) • • (=>) soundness – TSP does not find bugs in correct programs (<=) completeness – TSP detects all incorrect programs Apply Hypothesis H 0 H 1 … T 0 T 1 … Hk-1 Hk Tk-1 Tk Reduce Test set H => (P╞ SP <=> P╞o TSP, H) Keeping soundness and completeness depends on the hypothesis on the program 12

State of the Art Running Example • ATM System with the operations – login(password) / logged, wrong. Pass, blocked – logout – withdraw(amount) / give. Money(amount), not. Enough. Money • Following the second wrong login no more operations are allowed • The ATM distributes 20 or 100 CHF bills • Initial state – There are 100(CHF) in the account – ‘a’ is the right password, ‘b’ and ‘c’ are wrong passwords 13

State of the Art BGM and CO-OPN/2 14

Presentation Structure • • • Introduction State of the Art Problem Statement SATEL Discussion 15

Problem Statement • Adapt the previous theoretical work on BGM and CO-OPN/2 to test engineering • Build tools that can assist the test engineer in an industrial context 16

Presentation Structure • • Introduction State of the Art Problem Statement SATEL (Semi-Automatic TEsting Language) – Test Intentions – SATEL semantics – CO-OPN/2 c++ • Discussion 17

SATEL What are Test Intentions? < 4 x login. Logout A test intention defines both a subset of the SUT behavior and hypothesis about how to test it 1 withdraw reach. Blocked 18

SATEL Test Intentions and BGM Inspired from BGM and the work of Péraire, but we produce and reduce an exhaustive test set per test intention 19

SATEL Recursive Test Intentions and Composition • One test intention is defined by a set of axioms: Variables f : HML T in login. Logout; Base case for the recursion Recursive (empty testdefinition intention) f in login. Logout => f. HML({login(a) with logged>} {logout} T) in login. Logout; • Test intentions may be composed: f in login. Logout & nb. Events(f) < 4 => f in 4 Less. Login. Logout Test intention Regularity reuse over execution path HML(T), true HML({login(a) with logged} {logout} {login(a) with logged} {logout} T), true 20

SATEL Uniformity Variables Axioms pass : password Uniformity predicate uniformity(pass) => HML({login(pass) with wrong. Pass} {login(pass) with blocked} T) in reach. Blocked; HML({login(b) with wrong. Pass} {login(b) with blocked} T), true HML({login(a) with wrong. Pass} T), false HML({login(b) with wrong. Pass} {login(a) with blocked} T), false 21

SATEL Regularity and sub. Uniformity Variables obs : primitive. Observation am : natural regularity Axioms sub. Uniformity predicate (am <= 140) = true, sub. Uniformity(am) => HML({login(a) with logged} ({withdraw(am) with obs} T) in 1 withdraw; HML({login(a) with logged} {withdraw(120) with not. Enough. Money T), true HML({login(a) with logged} {withdraw(80) with give. Money(80)} T), true 22

SATEL Test Intention Definition Mechanisms 23

Presentation Structure • • Introduction State of the Art Problem Statement SATEL (Semi-Automatic TEsting Language) – Test Intentions – SATEL semantics – CO-OPN/2 c++ • Discussion 24

SATEL Semantics of a test intention • Test intention unfolding (solve recursion) • Calculate the exhaustive test set – Replace all variables exhaustively – Generate oracles by validating against the model • Positive tests are annotated with true • Negative tests extracted from test intentions but having an impossible last event, annotated with false • Reduce the exhaustive test set by solving all predicates on the variables 25

SATEL Semantics Annotations • Annotations correspond the conditions in the model allowing a state transition • In CO-OPN an annotation is a conjunction of conditions reflecting the hierarchy and dynamicity of the model 26

SATEL Semantics Equivalence Class Calculation Variables obs : primitive. Observation pass : password (AADT) sub. Uniformity(pass) => HML(<login(pass) with obs> in login; C 1: correct password C 2: wrong password HML({login(a) with logged} T), true HML({login(b) with wrong. Pass} T), true HML({login(a) with logged} T), true HML({login(c) with wrong. Pass} T), true 27

SATEL Semantics Equivalence Class Calculation (cont) Variables path : primitive. Hml sub. Uniformity(path), nb. Events(path)<4 => path in all. Under 4; C 1: correct password C 2: wrong password C 3: one wrong login C 4: true C 5: not enough money C 6: enough money 28

SATEL Semantics Main Results • Complete formal abstract syntax and semantics • Exhaustive test set for a test intention • Formal proof of the capability of expressing a test intention for generating a test set that can establish the correctness of an implementation • Equivalence class calculation by transition system annotation, independent of operational unfolding • Reduction of both positive and negative test sets • Oracles for test intentions having simultaneous inputs 29

Presentation Structure • • Introduction State of the Art Problem Statement SATEL (Semi-Automatic TEsting Language) – Test Intentions – SATEL semantics – CO-OPN/2 c++ • Discussion 30

CO-OPN/2 c++ ATM Model annotation = context sync conditions (object sync condition, object id) pairs 31

CO-OPN/2 c++ Main Results • New version CO-OPN/2 c++ with abstract formal syntax and semantics • Revision of the abstract syntax regarding context modules • Direct semantics rather than transformational – Primitive treatment of gates used as observations in testing – Primitive treatment of contexts allowing hierarchy and modularity in testing • Clear semantics allowing annotation generation for the transition systems used as oracles 32

Presentation Structure • • • Introduction State of the Art Problem Statement SATEL Discussion 33

Discussion • Operational experiments with modified Prolog pointed the need for a better formalization • We have developed a theory and a language for test engineering in the context of CO-OPN – Syntactically and semantically well founded – SATEL’s semantics is generic and can be applied to other event-based specification languages 34

Discussion (cont. ) • Developed an IDE for SATEL’s concrete syntax integrated with Coopn. Builder • A case study with an industrial partner (CTI) allowed beginning to identify methodological capabilities of SATEL 35

Discussion Future Work Theoretical Model Practical Test Generation • Formalization of decomposition hypothesis • Implementation of theoretical model using the DD / SDD approach • Quantification of coverage • Model-checking techniques 36

Discussion Outcome • Book Chapters: – – Levi Lúcio, Marko Samer: “Technology of Test-Case Generation” Model-Based Testing of Reactive Systems 2004. Published by LNCS Didier Buchs, Luis Pedro, Levi Lúcio: “Formal Test Generation from UML Models” Research Results of the DICS Program 2006. Published by LNCS • Conferences and Workshops: – – – Levi Lúcio, Luis Pedro, Didier Buchs: “A Methodology and a Framework for Model-Based Testing” RISE 2004. Published by LNCS Levi Lúcio, Luis Pedro, Didier Buchs: “A Test Language for CO-OPN Specifications” IEEE International Workshop on Rapid System Prototyping 2005. Proceedings Levi Lúcio, Luis Pedro, Didier Buchs: “Semi-Automatic Test Case Generation from CO-OPN Specifications” MTOOS 2006. Microsoft Research Technical Report • Talks – – – Several talks in the context of the past and present projects Invited keynote talk at the RISE workshop, Geneva, September 2006 Invited talk at the ETH Zurich, January 2006 Representation of the SMV group in an expert evaluation of the department, Geneva, October 2006 Invited talk (speaker Didier Buchs) at the SQS (Software Quality Conferences), Geneva, May 2008 37

Discussion Outcome • Past projects: – – VALID: Interreg, France/Swiss regional development. With Leirios technologies and CTI (Centre des Technologies de l’Information de l’état de Genève) Ve. DISS: DICS initiative, Hasler Foundation • Present and submitted projects – – COMEDIA: Man. Com initiative, Hasler Foundation TEST_INDUS: Interreg, France/Swiss regional development. With Leirios technologies and CLIO (industial partner) • Inside and outside the SMV group – – – Collaboration with Steve Hostettler in the direction of finding efficient operational techniques for SATEL’s semantics Collaboration with Andrei Berlizev in order to automate testing for fault tolerant system expressed in the CORA language Perspectives for the application of SATEL for testing in the context of DSLs provided by Ang Chen, Luis Pedro, Matteo Risoldi and Vasco Amaral’s group in Lisbon 38