Easy and Hard Ways to Reach Coverage Closure

Easy and Hard Ways to Reach Coverage Closure Avi Ziv Simulation-based Verification Technologies IBM Haifa research Lab IBM Labs in Haifa © 2010 IBM Corporation

IBM Labs in Haifa Scope of This Talk Verification Plan Directives Checking, Assertions Biased-Random Stimuli Generator Test Design Under Verification Simulator Test Pass Fail Coverage Information Coverage Reports 2 Coverage Analysis Tool © 2010 IBM Corporation

IBM Labs in Haifa The Truth About Coverage Analysis ³ The main goals of the coverage process are: ² Monitor the quality of the verification process ² Identify unverified and lightly verified areas ² Help understanding of the verification process ³ This leads to conflicting goals ² Want to collect as much data as possible ± So we do not miss important events ² User needs concise and informative reports ± So we do not drown in too much detail ³ Coverage analysis help to close the loop from coverage measurement to the verification plan and verification environment adaptation 3 © 2010 IBM Corporation

IBM Labs in Haifa The Truth About Coverage Analysis ³ Normal projects contains thousands or even millions coverage events ² Even with high coverage this means many hundreds (or thousands or millions) uncovered events ³ We never have enough resources to deal with all of them ³ Need to. Ints extract important information of them First Result RM Rnd out Count Last Fadd 0 Y 2 3/4/07 +0 - 0 + Y N 2 fsqrt q. Na. N Near Y 0 - - + - -Norm Near Y +Norm + Y 21 3 3/4/07 6/6/07 Fsub 1/6/07 14/2/07 6/6/07 9/6/07 24/4/07 +Norm + 21 1/6/07 9/6/07 Fadd +0 0 Y 2 3/4/07 6/6/07 Fadd +Norm + Y 21 1/6/07 9/6/07 - 3 14/2/07 24/4/07 fadd fmul fdivs N Y fsub - + + N fdivs -Norm Near Y fsqrt q. Na. N Near Fsub - + - - N Near -0 Near Fsub fsqrt fmul fnabs Fadd fmul fsqrt fmul fdivs fnabs q. Na. N -0 +0 +Norm q. Na. N + -Norm + Near - Near 0 + + Near N Y N 9 1 3 0 9 31/1/07 22/2/07 14/2/07 - 31/1/07 20/6/07 22/2/07 24/4/07 - 20/6/07 1 22/2/07 Y 11 0 3/4/07 6/6/07 21 - 11/5/07 2 1/6/07 9/6/07 N 9 3 14/2/07 9 31/1/07 20/6/07 Y 1 22/2/07 N 11 3/4/07 11/5/07 N Y Y Y N 0 - - 24/4/07 - 20/6/07 fdivs Fadd -Norm +Norm Near + Y Y 1 21 22/2/07 1/6/07 22/2/07 Fsub - + N 3 14/2/07 24/4/07 Fadd +0 0 Y fnabs fmul -0 + Near - N fdivs -Norm Near fnabs -0 Near Fadd +0 Fadd 2 3/4/07 6/6/07 9/6/07 11 9 3/4/07 31/1/07 11/5/07 Y 1 22/2/07 N 11 3/4/07 11/5/07 0 Y 2 3/4/07 6/6/07 +Norm + Y 21 1/6/07 9/6/07 Fsub - + N 3 14/2/07 24/4/07 fsqrt q. Na. N Near Y 0 - - fmul + - N 9 31/1/07 20/6/07 fdivs -Norm Near Y 1 22/2/07 Fadd +0 0 Y 2 3/4/07 6/6/07 Fadd +Norm + Y 21 1/6/07 9/6/07 fsqrt 4 +0 fadd Fadd q. Na. N Near Y N 0 - - 20/6/07 © 2010 IBM Corporation

IBM Labs in Haifa (Too) Detailed Status Reports ³ Detailed status reports can provide too much details even for a moderate coverage models ² Hard to focus on the areas in the coverage model we are currently interested in ² Hard to understand the meaning of the coverage information ³ Solution – advanced coverage analysis techniques ² Allow the user to focus on the current area of interest and look at the coverage data with the appropriate level of detail ² Two basic operations ± Select important events ± Group events together ² Three analysis techniques ± Manual analysis – coverage views and navigation ± Automatic analysis – hole analysis and quasi-holes ± Semi-automatic analysis – hole queries 5 © 2010 IBM Corporation

IBM Labs in Haifa Manual Analysis ³ Goals ² Allow the user to focus on the current area of interest and look at the coverage data with the appropriate level of detail ² Provide means for navigating between coverage reports to extract the useful information ³ Solution – coverage views ² Dynamically define the events to look at and granularity of the report 6 © 2010 IBM Corporation

IBM Labs in Haifa Projection ³ Project the n dimension coverage space onto an m (< n) subspace ³ Allow users to concentrate on a specific set of attributes ³ Help in understanding some of things leading up to the big picture 7 Instruction Count Density fadd 12321 127/136 fsub 10923 122/136 fmul 4232 94/136 fsqrt 13288 40/56 fabs 9835 38/40 © 2010 IBM Corporation

IBM Labs in Haifa Automatic Coverage Analysis ³ Detailed status reports do no always reveal interesting information hidden in the coverage data ² You need to know where to look at ² You need to know which questions to ask the coverage tool ³ Specifically, it is hard to find large areas of uncovered events in the coverage model 8 © 2010 IBM Corporation

IBM Labs in Haifa Large Holes Example ³ All combinations of two attributes, X and Y ² Possible values 0 – 9 for both (100 coverage events) ³ After a period of testing, 70% coverage is achieved Uncovered events X 0 0 1 Y 2 3 2 X 4 5 5 Y 4 2 8 X 7 7 7 Y 6 7 8 1 4 6 2 8 2 2 3 3 4 1 2 6 2 7 2 6 6 6 7 7 7 6 7 8 2 3 4 8 8 9 9 6 7 8 9 2 9 Y 2 D Visualization 9 8 7 6 5 4 3 2 1 0 0 1 2 3 4 5 6 7 8 9 9 X © 2010 IBM Corporation

IBM Labs in Haifa Finding Large Holes ³ 2 D visualization can be useful, but only in a limited number of cases ² Handling spaces with higher dimensionality is difficult ² Handling attributes with large number of values is difficult ² Handling unordered values is difficult ± Finding non-trivial patterns is difficult ³ Automatic techniques can overcome these problems ² Try to find large areas in the coverage space that are not covered ² Use basic techniques to combine sets of uncovered events into large meaningful holes 9 8 7 6 5 4 3 2 1 0 0 1 2 3 4 5 6 7 8 9 10 © 2010 IBM Corporation

IBM Labs in Haifa Notes on Hole Analysis Algorithms ³ Handling irrelevant (not interesting or not legal) events makes the algorithm conceptually more complex ³ What If The Hole Is Not Pure? ² Hole analysis produces large set of small holes ² There is a large area that is lightly covered ± This area can be more significant than the small holes Covered Uncovered Irrelevant 11 © 2010 IBM Corporation

IBM Labs in Haifa Automatic Analysis and Adaptation ³ Analysis of coverage data and adapting the verification plan and process according to them is one of the main bottlenecks of the verification process ² Need to handle huge amount of data ² Process is tedious and time consuming ² Requires expertise to: ± Identify important pieces of information ± Understand the root causes for them ± Help fix these root causes ³ Motivation Coverage analysis tools can assess the quality of the verification process, but not recommend how to improve it ³ Objectives Introduce an automatic mechanism to tune stimulus generation ² Stimulate hard-to-reach coverage points ² Improve rate of coverage ² Control coverage space distribution 12 © 2010 IBM Corporation

IBM Labs in Haifa Closing The Loop from Coverage to Stimuli ³ The problem: Given a coverage event that we want to hit, how to create a stimuli that reaches the requested event ³ In general, this is a very hard problem to solve because of the possible big distance between stimuli and coverage ² In terms of abstraction ² In terms of languages ² In terms of time 13 © 2010 IBM Corporation

IBM Labs in Haifa How to Hit Uncovered Coverage Events ³ The manual approach ² Based on understanding of the DUV and its environment ± Understand the target event and how to reach it ± Design the stimuli that reaches the event ² Break the problem into smaller problems ± Solve each problem separately ± Combine solutions ² Measure quality of solutions ± Iterate and improve until target reached ³ Automatic solution schemes basically follow the same approaches 14 © 2010 IBM Corporation

IBM Labs in Haifa Model-based Coverage Directed Generation ³ The basic idea ² Create a model of the DUV and query it on how to reach the target event ³ Model requirements ² Simple ² Accurate ² Supporting queries ³ An important and often difficult part of the solution is translation of the abstract test provided by the model into a concrete one 15 © 2010 IBM Corporation

IBM Labs in Haifa (Conceptual) Example ³ Build an abstract state machine that ² Emulate the operation of the DUV ² Has the target coverage event as a state or transition ³ Use model checker to find a path from the initial state to the target state ² By challenging it to proof that the state is unreachable ³ Convert the path into a concrete test process 1 0 request 4 16 2 request 6 5 3 © 2010 IBM Corporation

IBM Labs in Haifa Examples ³ Actually, I am not going to give specific examples ³ There are many papers published with the same basic idea. They present innovation in ² How to build the model ² How to traverse the model ² How to convert the abstract traversal to a concrete test ³ Most of this work is coming from academia and is working on small examples ² This approach is not adapted in industry 17 © 2010 IBM Corporation

IBM Labs in Haifa The Model Is Everything ³ If the model is accurate this approach works very well ² It is (almost) guaranteed to generate tests that reach the target events ³ But small inaccuracies can lead to big degradation in performance ³ Building and maintaining an accurate model can be a big problem ² Endless number of end cases to take care of ² Constant changes to the DUV 18 © 2010 IBM Corporation

IBM Labs in Haifa Exceptions ³ Automatic construction of the model from the implementation ² For example, by ignoring some of the state variables ² Here, the big problem is converting the abstract test into a concrete one ± Similar to abstraction-refinement in formal verification ³ Another possible exception are tools such as Trek by Breker and in. Fact by Mentor Source: Breker 19 © 2010 IBM Corporation

IBM Labs in Haifa Data-Driven Coverage Directed Generation Directives Biased-Random Stimuli Generator Test DUV Simulator Pass Fail Coverage Information CDG Engine Coverage Analysis Tool ³ Replace the power of the accurate model of model-driven CDG with the ability to learn/adapt based on observed data of stimuli (or directives) and the resulting coverage 20 © 2010 IBM Corporation

IBM Labs in Haifa How Data-Driven CDG Works ³ The CDG engine is fed pairs of inputs (directives) and outputs (coverage data) ² These pairs are often called training data ³ The CDG engine “understands” the relations between inputs and outputs and can answer queries about the relations ² What directive can lead to a requested coverage event? ³ Two levels of understanding ² Memorizing ² Generalization ³ In CDG we are usually interested in pairs not seen in the training data ² Specifically, how to reach uncovered events è Generalization is the key to success 21 © 2010 IBM Corporation

IBM Labs in Haifa How to Generalize ³ Need to know the relations between items in the output space ² And similarly in the input space ³ Example – ordering rules (<, >, =) ³ Example – similarity ² Usually means breaking the item into sub-items 22 © 2010 IBM Corporation

IBM Labs in Haifa Cross-Product Coverage and Generalization ³ Cross-product coverage is a natural form for generalization in the coverage space ² Break up the output space along the attribute’s axis ² Understand the input-output relations for each attribute ² Generalize by combining the understandings ³ But life is not that simple ² Attributes are related ² Conflicting understanding ² Randomness ² … 23 ? © 2010 IBM Corporation

IBM Labs in Haifa CDG Using Bayesian Networks ³ Model the CDG process rather than the design under test ³ Cast CDG as a statistical inference framework ³ Use Bayesian networks to represent relations among the CDG ingredients ² A natural and compact representation of the distribution space ² Enables encoding of essential domain knowledge cp_cmd_enable= [][] // }mode weight , {, 0 x 8, 30 -35 } {, 0 x. E, 1 -10 } ; { cp_core_enable= [][] // }mode weight { 0 x 2, 10 -100, }, {, 0 x 3, 10 -100 } ; { 24 CP Core Pipe Cmd 0 0 E * 1 0 18 2 0 * 1 D © 2010 IBM Corporation

IBM Labs in Haifa Bayesian Networks – Compact Representation of Probability Distributions via Conditional Independence Family of Alarm Qualitative part: Earthquake Burglary Directed acyclic graph (DAG) ³ Nodes – random variables ³ Edges – direct influence Radio Together: Define a unique distribution in a factored form 25 Alarm Call E B P(A | E, B) e b. 9 . 1 e b. 7 . 3 . 8 . 2 e b. 01. 99 Quantitative part: Set of conditional probability distributions © 2010 IBM Corporation

IBM Labs in Haifa Employing Bayesian Networks for CDG Directive Test Generator Simulator Coverage Tool Coverage Space Directive Space D 1 C 1 D 2 C 2 Mapping H 1 Dm 26 Coverage Cn © 2010 IBM Corporation

IBM Labs in Haifa Reaching a Specific Coverage Point 1. Map the point to values in the coverage nodes of the Bayesian network C 1=c 1, C 2=c 2, …, Cn=cn. 2. Query the network for the most probable explanation, (d 1, …, dm) = argmax P(C 1=c 1, …, Cn=cn | D 1, …, Dm) 3. Map the values in the directive nodes to test the directive file Test Directive File Specific Point D 1 C 1 D 2 C 2 Mapping H 1 Dm 27 Coverage Space Cn © 2010 IBM Corporation

IBM Labs in Haifa Advantages and Disadvantages ³ There are other similar approaches ² For example, the use of Inductive Logic Programming (ILP) to learn the relations between the stimuli and coverage (Hsueh and Eder) ³ All of them are less dependent on the accuracy of the model ³ But they do not guarantee to hit the target event ² At best they improve the probability of doing so ³ These approaches rely on some structure in the coverage model ² As is, they cannot work on singular coverage events 28 © 2010 IBM Corporation

IBM Labs in Haifa Back to The Shower ³ In model-based CDG, the model provides us an accurate solution ² But we depend on the accuracy of the model ³ In the data-driven approach we showed, we do not need an accurate model ² But all we get is improved probability of hitting the target ³ Yet another approach is to take existing attempts and iteratively improve them until the target is hit ³ Several such CDG systems exist based on ² Genetic algorithms ² Reward functions ² Path tracing 29 © 2010 IBM Corporation

IBM Labs in Haifa Genetic Algorithms ³ Algorithmic framework that tries to imitate nature evolution ³ There are several published papers on CDG using GA ² Mostly for processor verification (stimuli is assembly programs) ³ The basic idea 1. Choose the initial population of individuals 2. Evaluate the fitness of each individual in that population 3. Repeat on this generation until termination: 1. Select the best-fit individuals for reproduction 2. Breed new individuals through crossover and mutation operations to give birth to offspring 3. Evaluate the individual fitness of new individuals 4. Replace least-fit population with new individuals ³ The good and bad about GA is that we do not need to understand why changes improve the next generation, just to know that they do 30 © 2010 IBM Corporation

IBM Labs in Haifa Source: Nusym Path Tracing ³ Collect information on decisions made during simulation runs ² In the design and the testbench ³ Identify contributors to the decisions ³ Trace the contributors back to their roots ² For example, random decisions by the generator ³ Modify the roots to reach desired decisions ³ Properties ² Not guaranteed to find satisfying path ± Finds different path ² Scalable 31 © 2010 IBM Corporation

IBM Labs in Haifa Source: Nusym Path Tracing Example Random Run Path Tracing a = $random; a = 12; a = 24; b = $random; b = 21; c = a + b; c = 33; c = 45; if (c == 45) => else F b = 21; => then T d = 1; d = 0; d = 1; If (d == 0) T d = 0; => else => then $display(“HIT!!”); 32 Replay HIT!! © 2010 IBM Corporation

IBM Labs in Haifa Source: Nusym Example of CDG Results coverage Conventional simulation Nusym grading 100% IBM CDG ³ Base coverage ² 90% coverage after 55 K runs ³ CDG results ² >95% coverage after 4 K runs ² Two large holes identified 33 0% Lack of coverage because ³ TB over-constrained (40%) ³ Unreachable code (10%) ³ Dead code (40%) ³ Tool timeout (10%) © 2010 IBM Corporation

IBM Labs in Haifa Summary ³ Getting to coverage closure is one of the most difficult and time consuming tasks verification engineers face ³ The task has two important aspects ² Extracting important information out of the ocean of data ² Act upon this information to fix issues in activation of the verification environment ± E. g. , hit uncovered events ³ Advanced techniques and automation can help in both aspects ³ We are far away from having an end-to-end working solution ² But we are making progress 34 © 2010 IBM Corporation

IBM Labs in Haifa 35 © 2010 IBM Corporation