HWSW Codesign Verification and Debugging HW versus SW

HW/SW- Codesign Verification and Debugging. HW versus SW Ondrej Cevan

Outline what is testing embedding testing in design process of HW & SW testing tools in HW, SW Simulator, Emulator, Monitor

What is Testing? “to place a product or piece of equipment under everyday and/or extreme conditions and examine it for its durability, etc. ” (wiktionary. org) with testing we want to achieve: the object meets its specification estimation for reliability save cost (maintenance) improve quality debugging: the activity of locating, analyzing and removing a bug (an error state) from the system. HW/SW Codesign 16. 5. 2007 Verification and Debugging 3

Design Flow and Testing V-model in SW design assigns testing activities to development activities (figure from Yilin's slide) HW/SW Codesign 16. 5. 2007 Verification and Debugging 4

Design Flow and Testing model in HWdesign extension of every test process in V-model Simulations are checks for the designer to make sure correct behavior is being implemented we need to simulate the HW-clock HW/SW Codesign 16. 5. 2007 Verification and Debugging 5

Abstraction Levels SW: System Specification Language High-level Language Assembler Computer Instructions HW/SW Codesign 16. 5. 2007 HW: System Level Algorithmic Level Register Transfer Level Logic Level Circuit Level Chip Verification and Debugging 6

HW-model Behavior Simulation verifies the correctness of the HDL description Functional Simulation verification after synthesis, design with internal components Pre Layout Simulation not every code is synthesis-able after technology mapping, design with components of the target technology Post Layout Simulation to verify the results of place and route process timing and functionality Verification of the design checked and Debugging HW/SW Codesign 16. 5. 2007 7

Verification Methods (HW/SW) Review (of documents & code) Formal Verification (model checking, theorem proving) Dynamic Verification: functional testing , structural testing, performance, reliability, security testing simulations, black-box, white-box, code coverage, path coverage, throughput, resource usage, response time, stress/load testing, fault injection, simulation of attack. . . HW/SW Codesign 16. 5. 2007 Verification and Debugging 8

Verification Methods/Tools HW Hardware Implemented Fault Injection (HIFI) Hardware in the Loop Remote debugging tools (in-circuit emulator (ICE) = on-circuit debugger (OCD)= background debug module (BDM)) Logic Analyzer Oscilloscope (manual I/O testing) Simulation-based fault injection FPGA Prototyping (inject faults in a simulation model- on electrical level, gate level, register level, VHDL level, system level) HW/SW Codesign 16. 5. 2007 Verification and Debugging 9

SW/HW-Simulator simulator resembles real-world environment Hardware in the Loop, Software in the Loop, Model in the Loop Testbench (generates stimuli- input vector, emulates test cases) test case created from the specification document HW/SW Codesign 16. 5. 2007 Verification and Debugging 10

SW/HW-Debugger an analyze tool for developers to test and debug other programs program or fraction of code is processed line by line one can stop the program, read the values of variables and set new values to the variables certain test environments, test situations can be enforced also good for failure reproduction and fault analysis examples: GNU debugger (gdb), dbx, vhdldbx, Eclipse HW/SW Codesign 16. 5. 2007 Verification and Debugging 11

SW/HW- Emulator ability of a program or device to imitate other program or device in theoretical sense, Church-Turing thesis implies that any operating environment can be emulated within any other (in practice it can be quite difficult to achieve it) enables to test code for specific HW on a common PC we can emulate HW faults in the design phase (e. g. with fault injection) also SW faults can be emulated (Ballista tool) other ex. : wine, Commodore 64 emulator, . . . HW/SW Codesign 16. 5. 2007 Verification and Debugging 12

HW-Emulator Simulation: executes RTL code serially, good debug environment, user interface, easy, flexible, low cost, BUT: not fast enough for large designs Prototype: executes fully in parallel, fast, BUT needs logic analyzer to analyze limited number of signals, little to no debug capability HW-Emulator: rich debug environment, parallel execution, provides many features that can be found in logic simulators, a specific case of HW Emulator: In-Circuit Emulator HW/SW Codesign 16. 5. 2007 Verification and Debugging 13

Monitor software tool or hardware unit acts concurrent with the system or component observes /monitors, verifies, analyses and records the execution of the system e. g. measures the response time depending on the amount of user transactions (performance testing) HW/SW Codesign 16. 5. 2007 Verification and Debugging 15

When do we have tested enough? exhaustive testing is infeasible in practice we define coverage criteria that have to be met by a testing functional coverage (black-box view) structural coverage (white-box view) code coverage: path coverage, statement coverage, decision coverage, MCDC. . . data flow coverage in industry safety standards are used that describe the testing procedure and also the system development (RTCA/DO-178 b, IEC 61508), certification is also used in some domains HW/SW Codesign 16. 5. 2007 Verification and Debugging 16

Effort and Time many factors influence the time spent on testing know-how, experience, test-object (is it new to the test team? ), test strategy, requirements on the object: is it safety critical? , how intensive must the object be tested? , what quality of the object should be achieved? . . . in theory in SW the test effort should be approximately 50% of the total development effort (rule of thumb according to the book Basiswissen Softwaretest) in HW it should be approximately 70 -80% (according to the slides from VL Hardware Modeling) in practice: much less 10 -20% (according to the slides VL Hardware Modeling) HW/SW Codesign 16. 5. 2007 Verification and Debugging 17

Summary HW: verification needs HW an SW tools more effort is needed: no possibility for patches, the whole system/component must be replaced in case of failure SW: SW tools for verification are sufficient no need for extra testing hardware (oscilloscope. . ) HW/SW Codesign 16. 5. 2007 Verification and Debugging 18

List of used sources slides: VO Testing of Embedded Systems (Raimund Kirner, Astrid Ademaj) slides: VL Hardware Modeling (Martin Delvai) slides: VO Embedded System Engineering (Wilfried Elmenreich) book: Basiswissen Softwaretest, Andreas Spillner and Tilo Linz wikipedia. org wiktionary. org HW/SW Codesign 16. 5. 2007 Verification and Debugging 19