Tor X Automated Model Based Testing with Formal
Tor. X Automated Model Based Testing with Formal Methods © Jan Tretmans Radboud University Nijmegen (NL) together with: University of Twente Enschede (NL) tretmans@cs. ru. nl brinksma@ewi. utwente. nl Jan Tretmans Radboud University Nijmegen
Tor. X : Automated Model Based Testing with Formal Methods Contents F Model based testing F Formal, model based testing with transition systems ¨ Transition systems testing and ioco F A Tool for transition systems testing ¨ Tor. X F Current and future research F What does it mean for ARTIST 2 © Jan Tretmans Radboud University Nijmegen 2
Automated Model Based Testing test TTCN test generation cases tool model IUT conf model exhaustive IUT conf model sound test execution tool IUT passes tests © Jan Tretmans Radboud University Nijmegen pass fail 3
Model Based Testing F Testing with respect to a (formal) model / specification ¨ SDL, CSP, Lotos, Promela, UML, state diagrams, Spec#, . . F Precise, formal definition of correctness ¨ good and unambiguous basis for testing F Formal validation of tests F Algorithmic derivation of tests ¨ tools for automatic test generation F Maintenance of models / specifications, not of test suites ¨ regression testing © Jan Tretmans Radboud University Nijmegen 4
Model Based Testing with Transition Systems dertest : LTS generation (TTS) tool IUT conf i ioco model s exhaustive sound test execution t || tooli tool smodel LTS i IUT confs ioco model i IUT IOTS IUT i || passes der(s) tests pass © Jan Tretmans Radboud University Nijmegen pass fail 5
Formal Testing with Transition Systems Test hypothesis : s LTS der : LTS (TTS) ioco Proof soundness and exhaustiveness: Ts TTS IMPS i. IUT IOTS passes exec : : TESTS IOTS IMPS TTS {pass, fail} (OBS) © IUT IMP. i. IUT IOTS. t TTS. IUT passes t i. IUT passes t Jan Tretmans Radboud University Nijmegen i IOTS. ( t der(s). i passes t ) i ioco s pass / fail 6
Implementation Relation ioco Correctness expressed by implementation relation ioco: i ioco s =def Straces (s) : out (i after ) out (s after ) !x LU { }. p !x Straces ( s ) = { (L { })* | s p after = { p’ | p out ( P ) = { !x LU | p !x p © p = Jan Tretmans Radboud University Nijmegen } p’ } , p P } { | p p, p P } 7
Test Generation Algorithm To generate a test case from transition system specification s 0 compute T(S), with S a set of states, and initially S = s 0 after ; For T(S), apply the following recursively, non-deterministically: 1 end test case pass 2 supply input !a T( S after ? a ) © Jan Tretmans Radboud University Nijmegen 3 observe output forbidden outputs ? y allowed outputs ? x fail T ( S after !x ) allowed outputs or : !x out ( S ) forbidden outputs or : !y out ( S ) 8
Validity of Test Generation For every test t generated with algorithm we have: F Soundness : t will never fail with correct implementation i ioco s implies i passes t F Exhaustiveness : each incorrect implementation can be detected with a generated test t i ioco s © Jan Tretmans Radboud University Nijmegen implies t : i fails t 9
A Tool for Transition Systems Testing: Tor. X F On-the-fly test generation and test execution F Implementation relation: ioco F Mainly applicable to reactive systems / state based systems; ¨ specification languages: LOTOS, Promela, FSP, Automata user: manual automatic next input specification check output offer input Tor. X observe output IUT pass fail inconclusive © Jan Tretmans Radboud University Nijmegen 10
Tor. X Tool Architecture spec. explorer specification text © primer states transitions Jan Tretmans Radboud University Nijmegen Tor. X driver abstract actions adapter abstract actions IUT concrete actions 11
Tor. X © Jan Tretmans Radboud University Nijmegen 12
Tor. X Case Studies academic F Conference Protocol Philips F Easy. Link TV-VCR protocol Logica. CMG F Cell Broadcast Centre component Interpay F ‘’Rekeningrijden’’ Payment Box protocol Lucent F V 5. 1 Access Network protocol Logica. CMG F Easy Mail Melder academic F FTP Client Logica. CMG F “Oosterschelde” storm surge barrier-control ASML/Tangram F DO/DG dose control ASML/Tangram F Laser interface © Jan Tretmans Radboud University Nijmegen 13
What has been Achieved …… Sound and precise formal basis for model based testing ¨ ioco test theory ¨ proved test derivation algorithm Test tool Tor. X ¨ prototype tool for model-based formal testing ¨ “is at least as good as conventional testing” ¨ supports test generation and test execution ¨ more, longer, and provably correct test cases Applied successfully to different cases studies © Jan Tretmans Radboud University Nijmegen 14
Testing Transition Systems: Extensions Status model with data and time and hybrid and action refinement test case ? coin 1 ? coin 2 ? money n: int ! money ? coin 3 ? [ n 35 ] -> [ n 50 ] -> ? button 1 ? button 2 : = 00 c Vt: =: =00 Vcc : = d Vct </10 dt = 3 ! button 2 d Vcc</15 dt = 2 [Vt = 15 ] -> ! tea [[V c c =510 ] -> ! coffee ? tea pass © Jan Tretmans Radboud University Nijmegen fai l 15
Current and Future Research Twente & Radboud F Testing real-time aspects ¨ multi-channel real-time F Testing complicated data structures ¨ transformational- + transition system based testing F Action refinement ¨ when an abstract action is implemented as sequence of actions F What is a good test suite ¨ test selection and test coverage F Test adapter and test interface ¨ generic test environment F Compositionality and integration testing ¨ differences diminish © Jan Tretmans Radboud University Nijmegen 16
Current and Future Research Twente & Radboud F Hybrid testing ¨ when continuous variables occur F Compositionality and integration testing ¨ differences diminish F Testing stochastic and probabilistic properties F Multi-disciplinary ¨ system testing F Relations between model checking, testing, static analysis, theorem proving, etc. ¨ differences diminish F. . . © Jan Tretmans Radboud University Nijmegen 17
Some Dutch Testing Projects F Côte de Resyste (1998 - 2002) - Conformance Testing of Reactive Systems: Tor. X Philips Lucent TU Eindhoven Uni. of Twente F Atomyste (Logica. CMG) (Interpay) (KPN) - ATOm splitting in e. Mbedded s. YStem TEsting Uni. of Twente Radboud Uni. Nijmegen F Stress - Systematic Testing of Real-time Embedded Systems ¨ Testing real-time properties Uni. of Twente ¨ Testing data-intensive systems Radboud Uni. Nijmegen F Tangram - Model Based Testing and Diagnosis ¨ Testing ASML Wafer Stepper machines - application oriented ASML, ESI, TUD, TUE, UT, RU, S&T, TNO © Jan Tretmans Radboud University Nijmegen 18
ARTIST 2 Activities Quantitative Testing & Verification 1. Theory for testing embedded systems a. b. c. d. real-time aspects data aspects extended conformance testing theories integrating a. and b. test action refinement a. b. real-time schedulability analysis optimal control synthesis a. b. model checking algorithms for CTMC, MDP integration of performance analysis into verification a. b. data structures for real-time and stochastic modelling and analysis test interfaces and test adapters a. b. c. collection of case studies comparison identification of links to industrial tools 2. Verification and scheduling 3. Verification of stochastic systems 4. Tool-oriented research 5. Application of testing and verification tools in industrial settings © Jan Tretmans Radboud University Nijmegen 19
ARTIST 2 Activities Quantitative Testing & Verification 1. Theory for testing embedded systems a. b. c. d. real-time aspects data aspects extended conformance testing theories integrating a. and b. test action refinement a. b. real-time schedulability analysis optimal control synthesis a. b. model checking algorithms for CTMC, MDP integration of performance analysis into verification a. b. data structures for real-time and stochastic modelling and analysis test interfaces and test adapters a. b. c. collection of case studies comparison identification of links to industrial tools 2. Verification and scheduling 3. Verification of stochastic systems 4. Tool-oriented research 5. Application of testing and verification tools in industrial settings © Jan Tretmans Radboud University Nijmegen 20
- Slides: 20