Use of Verification for Testing and Debugging of

Use of Verification for Testing and Debugging of Complex Reactive Systems Mark Trakhtenbrot Holon Academic Institute of Technology

Why development of reactive systems is difficult ? Typically: q Critical applications (embedded RT controllers, …) q Must fulfill tough requirements (safety, timeliness, … ) Main problem: complex behavior q Intricate event-driven interaction with the environment; deep dependencies that are difficult to capture q Concurrency q Timing factors Result: design, analysis and validation are very difficult

Two approaches to system validation Testing - analyze behavior of the actual system, under selected test scenarios - system is examined in its real physical and operational environment Verification - “exhaustive testing” of system model under various assumptions on environment - ideally, fully certifies the system if property violated, generates the relevant test scenario

Testing - problems q q q Full testing is impossible Proper coverage; popular criteria (all states, all branches, all data flows) are important but not sufficient Good selection of tests is often based on tester’s experience and intuition In real systems: huge amount of tests is needed; manual creation is not realistic Tested property and test itself are often described in a non-formal, incomplete way - problem in interpretation of results - difficult to reproduce a failed run

Verification – challenges and limitations q q q Need to formalize system requirements Checked properties and environment assumptions are expressed in a formal assertion language; for example: - Temporal Logic to express order and timing constraints - templates for typical properties (response, precedence, …) Need to create a formal model of the actual system Problem: adequate abstraction from details that are not relevant to the checked property under-approximation: model violates the spec system violates the spec over-approximation: model fulfils the spec system fulfils the spec

q Verification limitations: - model need be final-state (restrict unbounded data, dynamically created objects, …) - exponential size of state space to be explored (challenge: find assumptions that are strong enough to narrow the search, while still covering interesting scenarios) - slow / unrealistic for big real-world models Testing: - no "finite state" limitation (may connect to real input sources, create unlimited amount of objects, …) - no limitations on system size - compiled code runs fast q

Combining T and V: how to achieve better results? q q q Typically: - testing first, to gain a good level of stability - verification for certification Model checking used for: - test generation (e. g. to achieve the state coverage: systematic check of state reachability, with scenario generation) - generate scenario leading to test’s initial configuration (satisfying the test’s pre-condition) Suggested: - reproduction of failed runs

Reproduction of failed runs q q Typically, only partial info related to the failed run is available: - the remembered one, or thought to be relevant - even when a trace available, it may be not detailed enough (decision on what to trace is taken in advance) A common problem, regardless the system type or reporter type (customer or tester)

Reproduction of failed runs q q Especially serious for reactive real-time systems: - even small changes in order / time of input events may have significant impact on system reaction - time-related factors are especially hard to reproduce - concurrency causes non-determinism in the execution order Challenge: how to utilize partial knowledge to reproduce and localize the error

Suggested process q Analyze the available knowledge about “irreproducible” system execution that violates property P - analysis of recorded trace - remembered observations q q q Use it to set up model checking of the system model that verifies whether P holds Translate the found model-level counter-example into a code-level scenario Run the code under this scenario and debug it

Considered framework: Model-based development q formal model of system (behavior, structure, …) q statecharts to capture behavior q q q executable model; a basis for automated tools (simulation, verification, code generation) model-level analysis while code execution (backward traceability) since code is derived from the model, they are (hopefully) always in synch

Model-based analysis with Statemate Simulation: check, record and replay test scenarios Verification: Model Checker / Model Certifier (developed by W. Damm & Co) - racing and non-determinism - reachability (drive-to-property) analysis - check of temporal properties (expressed with predefined templates), including timing constraints - when a property violated, generates a simulation script to reproduce the failure

Important abstractions Both use simulated time SIM synch time (clock-driven systems): every step takes the same time SIM asynch time (event-driven systems): when in stable state, advance clock, read inputs & run till stabilization; zero-time chain reaction Basic assumption: system is fast enough to complete its reaction before arrival of next stimuli

Important abstractions Maximum parallelism (full synchronization): At each step, all enabled transitions in all active statecharts are executed simultaneously

Analysis with generated code q q Code synthesis: executable C (for host or target OS) derived from the model What can we gain from testing of derived code if it so closely relates to the model? Overcome limitations of simulation and verification - Real Time execution, get rid of SIM time assumptions - Models augmented with design attributes (tasks/events q mapping, …) * generate highly optimized code for realistic target environment * tasks run asynchronously; full synchronization only within every separate task These can not be addressed by either simulation or verification

Suggested process q q Analyze the available knowledge about “irreproducible” system execution that violates property P - analysis of recorded trace - remembered observations Use it to set up model checking of the system model that verifies whether P holds (used to define environment assumptions) Translate the found model-level counter-example into a task that runs in parallel with the code and feeds as needed: right inputs at right time points (use Code. Gen API) Debug and localize the problem in the model, based on code-to -model backward traceability

What can / can’t be achieved this way? q q q The approach improves the ability to reproduce bugs, but it doesn’t provide an ultimate solution E. g. under the simulated time assumption (“the system is fast enough …”) MC may conclude that not P is never reached, even though P is violated in real-time execution The opposite is also possible. Hence the approach requires that violation of P in model will be re-checked, under the found scenario, also in code.

Example: Rail Cross Control

Rail Cross Control (cntd. ) Trains - speed: constant 10 … 160 km/h; length: 1, 000 … 3, 500 meters - distance between two trains on the same track: at least 4000 meters. Barriers - can work properly or be damaged (then need be repaired) - normally, opened / closed within 5 sec. - should be closed 40 sec before a trains enters the cross - should be opened when a train leaves the cross, unless another train is 55 sec or less before entering the cross Lights - are absolutely reliable - modes: off, blinking, hold on

Safe Cross Property If there is a train in any of the cross sectors, then for each of the tracks: - either the track's barrier is closed - or its light holds on

Functional view of the system

Cross control

Barrier control

Safe Cross Property Safe_Cross == IN_CROSS 1 or IN_CROSS 2 ( (BAR 1_CLOSED or LIGHT 1_ON) and (BAR 2_CLOSED or LIGHT 2_ON) )

Testing the generated code Code is generated directly from the model (no design attributes used) Tested scenarios: A single train on some of the tracks : OK Two trains on the same track, with required minimum distance between them : OK Two trains on different tracks : - 2 nd entered when the 1 st already exited from the cross: OK - 2 nd entered at some point before the 1 st reached the cross: FAILURE

Defining the Model Checker analysis Type of analysis: Drive-to-property (not Safe_Cross) To restrict the search, used the following techniques: - Freezing variable values - Simple assumptions (boolean formulae over model elements) - Statechart-modeling of general (independent of specific scenario) facts about environment behavior Freeze: - TRAINi_SPEED : 160, TRAINi_LENGTH : 3500 (if they were not known, then had to properly define the variables’ ranges) - Barriers were not damaged, hence input events OUT_OF_USE* and REPAIR* are frozen at value false

Defining the Model Checker analysis Simple assumptions: Violation observed with only one train on each of the tracks: ENTER_TRACKi not (IN_STARTi or IN_CROSSi or IN_FINISHi) (i=1, 2) Train on the 2 nd track entered when there was a train on the 1 st track, and it didn’t reach yet the cross segment: ENTER_TRACK 2 IN_START 1 and not (IN_CROSS 1 or IN_FINISH 1)

Defining the Model Checker analysis Environment modeling

Model checking result and its use for code testing Model Checker: - found reachable configuration satisfying not Safe_Cross - created a script that drives simulation to that configuration Testing of generated code: - used Code. Gen API to translate the script into a driver-task that runs aside the system code and feeds it according to the found scenario - instrumented the code for creation of execution traces (used the relevant Code. Gen’s option) - run black-box test (traced in/out events); Safe_Cross failed - run white-box test (traced some internals), to localize the problem

Black-box testing -- at 0: 08. 020 : event ENTER_TRACK 1 generated -- at 0: 00: 24. 985 : event ENTER_TRACK 2 generated -- at 0: 00: 27. 198 : condition BAR 1_CLOSED changed value to true -- at 0: 00: 27. 248 : condition BAR 2_CLOSED changed value to true -- at 0: 00: 27. 287 : condition LIGHT 1_FLASHING changed value to true -- at 0: 00: 27. 331 : condition LIGHT 2_FLASHING changed value to true -- at 0: 01: 03. 801 : event ENTER_CROSS 1 generated -- at 0: 01: 04. 732 : event ENTER_FINISH 1 generated -- at 0: 01: 20. 765 : event ENTER_CROSS 2 generated -- at 0: 01: 21. 646 : event ENTER_FINISH 2 generated -- at 0: 02: 22. 544 : event EXIT_START 1 generated -- at 0: 02: 23. 455 : event EXIT_CROSS 1 generated -- at 0: 02: 27. 572 : condition BAR 1_CLOSED changed value to false -- at 0: 02: 27. 611 : condition BAR 2_CLOSED changed value to false -- at 0: 02: 27. 654 : condition LIGHT 1_OFF changed value to true -- at 0: 02: 27. 696 : condition LIGHT 2_OFF changed value to true

Conclusions q q The approach improves the ability to reproduce bugs that are otherwise hardly reproducible Applicable for more complex properties, such as ALWAYS (BAR_DOWN EVENTUALLY (10) in[CLOSED] or LIGHT_ON) Based on support by Model Certifier Limitations due to potential semantic differences between model and generated code; model checking results must be re-tested with generated code q q Test execution for derived code can be run-time verified using a monitor automatically created from the formal