Efficient Instrumentation for Code Coverage Testing Mustafa M

Efficient Instrumentation for Code Coverage Testing Mustafa M. Tikir Jeffrey K. Hollingsworth University of Maryland

Evaluation of Code Coverage l Measuring code coverage is important to – Identify unexecuted program statements – Verify that each path is taken at least once l Requires extensive instrumentation – Must determine if every statement is executed University of Maryland

Instrumentation for Code Coverage l Traditional Approach – Static instrumentation using counters – Instrumentation code remains for entire execution – Conservative instrumentation of all possibly needed instrumentation code for all functions l Useless instrumentation wastes time – especially for long running programs such as servers and enterprise software l Full instrumentation increases setup time University of Maryland

Our Approach l Insert instrumentation code dynamically – Pre-Instrument all functions at program start – On-Demand instrument at first function call l l Periodically remove instrumentation Use dominator trees – Reduces instrumentation within a function l Use incremental function instrumentation – Insertion of instrumentation on first call – Reduced which function gets instrumented University of Maryland

Dyninst API Modify code in a running program Implementations available for l l – Alpha, Sparc, Power, Mips and x 86 architectures A mutator program l – Generates machine code from high-level code – Transfers machine code to a running mutatee program l l Mutatee is the application being instrumented Base-trampoline contains – – l Program relocated instructions slots for calling code Mini-trampoline stores the inserted code University of Maryland Base Tramp Mini Tramp Pre Save Registers Setup Args Relocated Instruction Code Snippet Post Restore Registers

Using Dominator Trees l Definitions: – A dom B if all paths from entry to basic block B goes through basic block A – A idom B if, for all C, (C != A) and (C dom B) implies (C dom A) l C A Fact: – If a basic block, B, is executed • all basic blocks along the path from B to the root of dominator tree also execute University of Maryland B

Leaf Node Instrumentation l Leaf nodes in dominator tree are instrumented – Coverage of internal nodes will be inferred l Coverage information propagated from leaf nodes to entry Dominator Tree Control Flow Graph Entry 1 1 3 2 2 3 5 4 4 Exit University of Maryland 5

Non-Leaf Node Instrumentation l Leaf node instrumentation is necessary but not sufficient – Control flow might cause cross edges in dominator tree Control Flow Graph Dominator Tree Entry 1 2 3 2 1 Exit 3 Exit l We also instrument basic block, A, if l l A has at least one outgoing edge to basic block, B, and A does not dominate B University of Maryland

Code Coverage Algorithm Pre-Instrumentation l On-Demand Instrumentation At Start l – Create CFG and dominator trees for all functions – Instrument the basic blocks selected l – Insert breakpoints at each function entry l During Execution At Termination – Propagate and collect the coverage information During Execution – On breakpoint • Identify the function • Create CFG and dominator tree • Instrument the basic blocks selected – At fixed time intervals delete the executed code – Stop the execution at fixed time intervals – Delete the executed instrumentation code l At Start l At Termination – Propagate and collect the coverage information University of Maryland

Reduction In Instrumentation Points l 34 -49% with pre-instrumentation l 42 -79% with on-demand instrumentation University of Maryland

SPEC/compress Coverage Curve l Covers 76% in the first 18% of the execution – Most of the basic blocks that will execute are covered at the beginning of the program – Rest of the run is re-executions University of Maryland

SPEC/compress Execution Time l l l Purecov is a commercial state of art code coverage tool Our code coverage tools outperform purecov Significant reduction when dynamic code deletion is enabled – Most of the instrumentation code is deleted at the beginning – Setup time and deletion overhead is insignificant • A few number of basic blocks to instrument and check University of Maryland

Postgre. SQL Coverage Curve l Wisconsin benchmark queries – Measure the performance of database systems • Executes select/join queries repeatedly University of Maryland

Postgre. SQL Execution Time l Setup, Instrumentation and Deletion Time Using on-demand instrumentation – Our coverage tools outperform purecov almost always l On-demand instrumentation outperforms preinstrumentation University of Maryland

Instrumentation Execution Frequency l l l Overhead of our dynamic code coverage system is bursty Running previously unexecuted code results in the execution of a significant amount of instrumentation Running previously executed code does not result in the execution of any instrumentation University of Maryland

Overall Slowdown l l l Results for our code coverage tool are for 2 -second deletion Slowdown using purecov ranges from 1. 83 to 19. 78 Slowdown using our code coverage tools range – From 1. 002 to 2. 6 for on-demand instrumentation – From 1. 002 to 4. 96 for pre-instrumentation University of Maryland

Dyninst Coverage Tool University of Maryland

Conclusions l Dominator trees – Reduce instrumentation points by 34 -49% – Plus on-demand instrumentation reduce instrumentation points by 42 -79% l Combining dominator trees and on-demand instrumentation reduces – Setup time and deletion interval overhead – Runtime overhead by 38 -90% compared to purecov l Dynamic deletion of instrumentation – Computes coverage information faster University of Maryland

Conclusions (cont) l l Code Coverage overhead reduced to about 10% Code coverage can now be included as part of production code – Information about the execution of extremely infrequent error cases will be provided – Reduced overhead for residual testing l Dyninst library + Dyninst Coverage tools – http: //www. dyninst. org University of Maryland