Simulation based verification coverage Verification coverage estimates the
Simulation based verification: coverage Verification coverage estimates the quality of simulation Digitaalsüsteemide verifitseerimise kursus 1
Coverage metrics • System can’t be simulated with all possible stimuli • Question: how many stimuli are needed? • Coverage metrics used • Coverage shows our confidence in how thoroughly we verified Digitaalsüsteemide verifitseerimise kursus 2
Coverage metrics • Three types: – Code coverage – Parameter coverage – Functional coverage • Also will talk about – Domain coverage, cross coverage, stuck-at fault coverage Digitaalsüsteemide verifitseerimise kursus 3
Coverage metrics: code coverage • Code coverage shows how thoroughly is the code examined during simulation L Based on implementation only J Easy to compute Digitaalsüsteemide verifitseerimise kursus 4
Coverage metrics: parameter coverage • How well parameters and dimensions covered in functional blocks • Based on implementation, spec or both Digitaalsüsteemide verifitseerimise kursus 5
Coverage metrics: functional coverage • Based on behaviors derived from the spec • Shows the number of covered behaviors • Problem: how to formally generate the behaviors and guarantee that the full functionality is covered… Digitaalsüsteemide verifitseerimise kursus 6
Comparison: coverage metrics • None of the three types is universally best – Advantages of code coverage are simplicity and formalisation – Advantage of functional coverage is high verification power – Parameter coverage is a trade-off between the two • Coverage metrics may be used for guiding test generation • To reach 100 % coverage is very much harder than to reach 90 %. . . Digitaalsüsteemide verifitseerimise kursus 7
Code coverage: statement coverage • All lines except begin, end, else • Example: a > x, x = y • Statement coverage 8 out of 10 = 80% Digitaalsüsteemide verifitseerimise kursus 8
Code coverage: block coverage Digitaalsüsteemide verifitseerimise kursus 9
Instrumented code • Instrumented code applied in code coverage analysis. Thus, simulation slowed down considerably. • Turn off the code already covered by simulation • In that case almost no extra time penalty towards the end of simulation Digitaalsüsteemide verifitseerimise kursus 10
Instrumented code Digitaalsüsteemide verifitseerimise kursus 11
Code coverage: branch and path • More stringent than statement coverage • Path coverage does not scale Digitaalsüsteemide verifitseerimise kursus 12
Code coverage: expression coverage • Again, more stringent than statement coverage • We analyze which expressions were evaluated • Expressions divided into layers: E = (x 1 x 2 + x 3 x 4) layer 1: E = y 1 + y 2 layer 2: y 1 = x 1 x 2, y 2 = x 3 x 4 • With multi-layer expressions, either the top layer or all layers analyzed Digitaalsüsteemide verifitseerimise kursus 13
Code coverage: expression coverage • Minimum input tables applied: f = x 1 & x 2 0 0 1 x 1 0 1 x 2 0 1 f = (x > y) x > y meaning 0 0 x <= y 1 1 x > y f = (y ? x 1 : x 2) 1 0 y x 1 x 2 1 1 1 0 0 • Expression x 1 & x 2 and two input stimuli: (x 1=1, x 2=0) and (x 1=1, x 2=1) • In that case the expression coverage for & is 66. 67% Digitaalsüsteemide verifitseerimise kursus 14
Code coverage: state coverage • Sequence b, a, . . . gives 66. 67% of state coverage, as S 1 and S 2 traversed (2 out of 3). • S 4 is illegal state, thus max state coverage is 75%! Digitaalsüsteemide verifitseerimise kursus 15
Code coverage: transition coverage • Sequence b, a, . . . gives 40% of transition coverage as 2 out of 5 transitions traversed. • Transition S 2/b not defined! • Calculated by transitions specified (5) or all possible transitions (6) Digitaalsüsteemide verifitseerimise kursus 16
Code coverage: sequence coverage • User can specify state sequences at the basis of the coverage calculation • Normally essential functionality or corner cases specified • Possible to keep track on which desired (or undesired) sequences were simulated Digitaalsüsteemide verifitseerimise kursus 17
Code coverage: toggle coverage • Measures how many bits in the signals changed their state between 0 and 1 • E. g. a signal for a 4 -bit bus and values 0000 and 1101 would give a toggle coverage of 75% Digitaalsüsteemide verifitseerimise kursus 18
Parameter coverage Parameter Server configuration Task processing time Length of the queue Range {(S 1, S 2): S 1, S 2 passive or active} {3, . . . , 6} {(m 1, m 2, m 3): 1 mi 8, 1 i 3 } Digitaalsüsteemide verifitseerimise kursus 19
Parameter coverage: example Parameter Server configuration Task processing time Length of the queue Range {(S 1, S 2): S 1, S 2 passive or active} {3, . . . , 6} {(m 1, m 2, m 3): 1 mi 8, 1 i 3 } • If simulation provides 2 server configurations: (S 1 active, S 2 passive) ja (S 1 active, S 2 active); processing times 3, 4 and 5; maximum que lengths (6, 4, 8), then . . . • . . . Parameter coverages: server conf. 50 %, processing times 75 %, queue lengths 75%, 50%, 100% Digitaalsüsteemide verifitseerimise kursus 20
Functional coverage • Functional coverage based on behaviors derived from the specification • Functional coverage reports the ratio of covered behaviors • Problem: how to formally generate behaviors and guarantee that the entire functionality is covered Digitaalsüsteemide verifitseerimise kursus 21
Cross coverage • Cartesian product of 2 or more coverages • More stringent than using just a single coverage metric • Cartesian product of several coverages may represent a huge space Digitaalsüsteemide verifitseerimise kursus 22
Domain coverage (boundary coverage) • Every condition defines a partition of the input domain • Small design error may slightly shift the partitioning foo (in 1, in 2) if (in 1 < in 2) out = 1; else out = 0; out=1 in 2 out=0 in 1 • What if in 1 < in 2 should be rather in 1 <= in 2? • What if the value of in 1 or in 2 is slightly wrong? Digitaalsüsteemide verifitseerimise kursus 23
Domain coverage: Setting up a test • Testing with a value at the boundary and with a value immediately off the boundary foo (in 1, in 2) if (in 1 < in 2) out = 1; else out = 0; in 2 out=1 out=0 2 1 1 in 1 • Test 1: in 1, in 2 = 1, 1 – false, at the boundary • Test 2: in 1, in 2 = 1, 2 – true, immediately off the boundary Digitaalsüsteemide verifitseerimise kursus 24
Domain coverage: example • Per each condition (predicate) two tests: – True and immediately off the boundary – False and immediately off the boundary a = in 1 + in 2; b = 0; c = 0; while (c < a) c = c + in 1; if (c < in 2) out = a + b; else out = a + c; Test 1: in 1, in 2 = 0, 0 • c = 0, and a = 0 • false and at the boundary for c < a • false and at the boundary for c < in 2 Test 2: in 1, in 2 = 1, 0 • c = 0, and a = 1 • true and at the boundary for c < a • c set to 1 • false and at the boundary for c < in 2 • 3 conditions out of 4 satisfied, 75% domain coverage Digitaalsüsteemide verifitseerimise kursus 25
Issues with domain coverage • Non-linear boundaries Partly linear boundaries: test each part separately (in 2 > in 1) and (in 1 > 2) in 2 2 in 1 Higher order boundaries: no standard solutions (in 2 > in 12) in 2 in 1 Digitaalsüsteemide verifitseerimise kursus 26
Observation coverage • Code coverage ignores the observability aspect • It may happen that the covered code items have no influence to any observable point • It has been showed that 100% code coverage may result in 60 -80% observation coverage in the worst case [Fallah, Devadas, Keutzer DAC’ 98]. • Thus, we should consider propagating the bug effect to an observable output Digitaalsüsteemide verifitseerimise kursus 27
Other metrics: stuck-at coverage • Assume that a bug manifests itself as a signal line being constantly stuck to 0 (or 1) • Assune single faults, i. e. only one signal line is faulty at a time • Stuck-at coverage indicates how well we have excercised the structure Digitaalsüsteemide verifitseerimise kursus 28
Issues with coverage • None of the coverage metrics provides 100% of confidence in correct functionality • E. g. 100 % stuck-at coverage test for OR gate does not discover the design error OR XOR x 1 x 2 x 1 V x 2 x 1 x 2 x 1 x 2 0 1 1 1 0 1 0 0 0 = Digitaalsüsteemide verifitseerimise kursus 29
Issues with coverage • Similar problem with expression coverage • 100 % expression coverage for OR gate does not discover the error OR XOR x 1 x 2 x 1 V x 2 x 1 x 2 x 1 x 2 0 1 1 1 0 1 0 0 0 = Thus, we need to consider a set of coverages! Digitaalsüsteemide verifitseerimise kursus 30
- Slides: 30