Debugging with the Total View Source Code Debugger

Debugging with the Total. View Source Code Debugger MIT March 6, 2008 Ed Hinkel Sales Engineer Total. View Technologies

Agenda • Total. View Technologies Intro • Source Code Debugging - Setup - Navigation - Data View and Analysis • • 2 Memory Debugging Parallel Debugging Large Apps Questions / Comments

Total. View Technologies Corporate Overview • The Most Experienced Technologists in Parallel Debugging • Technology originally developed at BBN in late 80’s • Developed from scratch specifically for debugging parallel applications • Total. View is recognized worldwide as the gold standard for debugging in multi-core, data intensive, high-performance, distributed, and clustered computing environments • • The debugging leader in the HPC, EDU, and Commercial sectors Founded as Etnus, Inc. in 1999, Renamed Total. View Technologies in 2007 50 employees (heavily engineering influenced) Over 1, 400 customers in 55 countries Over 10 K developers with over 2 million cores under license • • 3 Award winning product line (Supercomputing Online's Product of the Year)

What Is Total. View? 4

What is Total. View? • A comprehensive debugging solution for demanding multi-core applications • C, C++, Fortran 77 & 90, UPC • Wide compiler & platform support • Multi-threaded Debugging • Parallel Debugging • MPI, PVM, Others • Remote Debugging • Memory Debugging Capabilities • Integrated into the Debugger • Powerful and Easy GUI • Visualization • CLI for Scripting 5

Supported Compilers and Architectures • • • 6 Platform Support • Linux x 86, x 86 -64, ia 64, Power • Mac Power and Intel • Solaris Sparc and AMD 64 • AIX, Tru 64, IRIX, HP-UX ia 64 • Cray X 1, XT 3, XT 4, IBM BGL, BGP, Si. Cortex Languages / Compilers • C/C++, Fortran, UPC, Assembly • Many Commercial & Open Source Compilers Parallel Environments • MPI (MPICH 1 & 2, LAM, Open MPI, poe, MPT, Quadrics, MVAPICH, & many others ) • UPC

Architecture for Cluster Debugging • Single Front End (Total. View) • GUI • debug engine • Debugger Agents (tvdsvr) • Low overhead, 1 per node • Traces multiple rank processes Compute Nodes • Total. View communicates directly with tvdsvrs • Not using MPI • Protocol optimization Provides Robust, Scalable and efficient operation with Minimal Program Impact 7

Total. View Basics _________ Startup, Process Control & Navigation

Starting Total. View Normal totalview [ tv_args ] prog_name [–a prog_args ] Attach to running program totalview [ tv_args ] prog_name –pid PID# [–a prog_args ] Attach to remote process totalview [ tv_args ] prog_name –remote name [–a prog_args ] Attach to a core file totalview [ tv_args ] prog_name corefile_name [ –a prog_args ] 9 Command Line GUI

Interface Concepts • Root Window • State of all processes being debugged • Process Window • Detailed state of a single process • Thread within a process • Point of control 10 • Control the process and possibly other related processes

Total. View Root Window Hierarchical/ Host name Linear Toggle Rank # (if MPI program) Total. View Thread ID # Expand - Collapse Toggle 11 Process Status Action Point ID number

Process Window Overview Toolbar Stack Trace Pane Stack Frame Pane Source Pane Tabbed Area 12

Stack Trace and Stack Frame Panes Language 13 Name Function Pointer

Source Code Pane 14

Viewing Source Code • TV always tries to display source code • If it cannot you will see assembly • -g puts ‘symbol table’ and ‘source code + line number’ info into your application • These are references, usually by relative path from the object file to source file • TV takes the basename and the path • Total. View will first try to use this info to find the source file • Then it will search a TV search path for the basename • Paths can be set via $tree function • CLI variables provides for setting source search paths - see documentation for details 15

Debugging Assembly Code 16 Display/Debug Source, Assembly or Both

Process Status 17

Stepping Commands Based on PC location 18

Basic Process Control Automatic Grouping • Control Group –All the processes created or attached together • Share Group –All the processes that share the same image • Workers Group –All the processes & threads that are not recognized as manager or service processes or threads • Lockstep Group –All threads at the same PC 19

Finding Functions, Variables, and Source Files Menu: View > Lookup -----Accelerator Keys: f, v -----“Closest Match” Search Results 20

Action Points Breakpoints -----Barrier Points -----Conditional Breakpoints -----Evaluation Points -----Watchpoints 21

Setting Breakpoints Setting action points Single left-click outlined source code line numbers Action Points Tab Lists all action points Dive on an action point to focus it in source pane Action point properties Context menu when right-clicking the action point Deleting action points Left-click in Source Pane Context menu in Source Pane / Action Points Tab Disabling action points Context menu Left-click in Action Points Tab 22 Saving all action points Action Point > Save All

Setting Breakpoints 23

Conditional Breakpoint 24

Evaluation Points • Generalization of Conditional Breakpoints • C/C++ or Fortran • Call functions • Set variables • Test conditions • Test small source code patches • Help set up program circumstances 25

Test Fixes on the Fly 26

Watchpoints Use Tools > Watchpoint from a Variable Window. Watchpoints are set on a fixed memory region. When the contents of watched memory change, the watchpoint is triggered and Total. View stops the program. Watchpoints are not set on a variable. You you need to be aware of the variable scope. Watchpoints can be conditional or unconditional Use intrinsic variables $newval and $oldval in the conditional expression 27

Using Set PC to Replay Code 28

Help System • Context sensitive buttons on many dialog windows • Help menu in the main windows • Launches an html browser • Navigate or search the full content • Also available in pdf and hard copy • Check out the tip of the week archive 29

Total. View Documentation 30

Total. View Basics _________ Viewing and Editing Data

Diving on Variables You can use Diving to: … get more information … to open a variable in a Variable Window. … to chase pointers in complex data structures. You can Dive on: … variable names to open a variable window … function names to open the source in the Process Window. … processes and threads in the Root Window. 32 How do I Dive? • Double-click the left mouse button on selection • Single-click the middle mouse button on selection. • Select Dive from context menu opened with the right mouse button

Diving on Variables 33

Undiving In a Process Window: retrace the path that has been explored with multiple dives. In a Variable Window: replace contents with the previous contents. 34 You can also remove changes in the variable window with Edit > Reset Default.

Dive in All displays an element in an array of structures as if it were a simple array. 35

The Variable Window • Window contents are updated • • automatically Changed values are highlighted “Last Value” column is available 36 Editing Variables • Click once on the value • • Cursor switches into edit more Esc key cancels editing Enter key commits a change Editing values changes the memory of the program

Expression List Window Add to the expression list using contextual menu with right-click on a variable, 37 or by typing an expression directly in the window

Expression List Window • • 38 Reorder, delete, add Sort the expressions Edit expressions in place Dive to get more info • • Updated automatically Expression-based Simple values/expressions View just the values you want to monitor

Four Ways to Look at Variables • Glance • Stack frame • Hover • Source pane • Dive to data window • Source, Stack or Variable Window • Arrays, structures, explore • Monitor via expression list • Source, Stack or Variable window • Keep an eye on scalars and expressions 39

Viewing Arrays 40

Slicing Arrays Slice notation is 41 [start: end: stride]

Filtering Arrays 42

Visualizing Arrays • Visualize array data using Tools > Visualize from the Variable Window • Large arrays can be sliced down to a reasonable size first • Visualize is a standalone program • Data can be piped out to other visualization tools • Visualize allows to spin, zoom, etc. • Data is not updated with Variable Window; You must re-visualize • $visualize() is a directive in the expression system, and can be used in evaluation point expressions. 43

Typecasting Variables • Edit the type of a variable • Changes the way Total. View interprets the data in your program • Does not change the data in your program • Often used with pointers • Type cast to a void or code type to snoop around in memory Give Total. View a starting memory address and Total. View will interpret and display your memory from that location. 44

Type Casts Read from Right to Left Examples: • • 45 int[10]* Pointer to an array of 10 int*[10] Array of 10 pointers to int*[10]* Pointer to an array of 10 pointers to int[5]*[10] Array of 10 pointers to arrays of 5 int

Typecasting Examples 46 • Cast float * to float [100]* to see a dynamic array’s values • Cast to built-in types like $string to view a variable as a null-terminated string (automatic cast for char *) • Cast to $void for no type interpretation or for displaying regions of memory • Cast to $code[100] to see 100 instructions of disassembly • Cast to your own structs, objects, Fortran user defined types, common block definitions, etc.

STLView transforms templates into readable and understandable information –STLView supports std: : vector, std: : list, std: : map, std: : string –See doc for which STL implementations are supported 47

C++ Templates Total. View understands your C++ templates and gives you a choice. . . Boxes with solid lines around line numbers indicate locations with replicated code

Managing Signals File > Signals 49 Error Stop Resend Ignore Stop the process and flag as error Stop the process Pass the signal to the target and do nothing: use with signal handlers Discard the signal

Total. View Basics _________ Memory Debugging

What is a Memory Bug? • A Memory Bug is a mistake in the management of heap memory • Failure to check for error conditions • Leaking: Failure to free memory • Dangling references: Failure to clear pointers • Memory Corruption • Writing to memory not allocated • Over running array bounds 51

The Agent and Interposition Process Total. View User Code and Libraries Heap Interposition Allocation Table Agent (HIA) Deallocatio n Table Malloc API 52

Total. View HIA Technology • Advantages of Total. View HIA Technology • Use it with your existing builds • • Programs run nearly full speed • • 53 No Source Code or Binary Instrumentation Low performance overhead Efficient memory usage Low memory overhead Support wide range of platforms and compilers

Memory Debugger Features • Automatically detect allocation problems • View the heap • Leak detection • Block painting • Dangling pointers • Deallocation/reallocation notification • Guard Blocks • Memory Comparisons between processes • Collaboration features 54

Heap Graphical View 55

Heap Graphical View 56

Leak Detection • Based on Conservative Garbage Collection • Can be performed at any point in runtime • Helps localize leaks in time • Multiple Reports • Backtrace Report • Source Code Structure • Graphically Memory Location 57

Leak Detection 58

Guard Blocks & Memory Corruption 59

Guard Blocks & Memory Corruption 60

Memory Comparisons • “Diff” live processes • Compare processes across cluster • Compare with baseline • See changes between point A and point B • Compare with saved session • 61 Provides memory usage change from last run

Memory Status 62

Memory. Scape • What is Memory. Scape? • • • Features • • • 63 Streamlined Lightweight Intuitive Collaborative Memory Debugging Shows • Memory Errors • Memory Status • Memory Leaks • Bounds Violations MPI Memory Debugging Remote Memory Debugging • • Tech • Low Overhead • No Instrumentation Interface • Inductive • Collaboration • Multi-process

Script Mode • Automation Support • Memory. Scape lets users run tests and check programs for memory leaks without having to be in front of the program • Simple command line program called memscript • Doesn’t start up the GUI • Can be run from within a script or test harness • The user defines • What configuration options are active • What things Memory. Scape is looking for • What actions Memory. Scape should take for each type of event that may occur 64

Total. View Basics _________ Parallel Application Debugging

Challenges of Debugging in a Multi-Core Age • Concurrency • Stochastic errors are often many times harder to solve than others • Achieving reproducibility of race conditions, deadlocks, live-locks, and other concurrent bugs is the key • Precise thread-level control of all the processes in the distributed application • If you can’t control the threads then you are simply hoping that the problem happens • If you can’t reproduce the problem you can’t easily pose questions about why • Constructs to enable problem reproduction • Scripting • Expression Evaluation • Visibility into all the relevant data 66 • Thread specific stacks and thread-private variables • Easy ways to view complex data

Debugging Multithreaded Programs When debugging multithreaded programs, you want to: • • Know where to look to get thread status. Be able to switch the focus from one thread to another quickly and • easily. Understand how asynchronous thread control commands (step, go, halt) and breakpoints are used. A parallel program has a lot more states than ‘Running or stopped’ There are more degrees of freedom for program control. Total. View gives you a full set of features to manage this complexity. It is important to understand how the different classes of commands and features work so as to avoid confusion. 67

Total. View Provides • • • 68 MPI-Aware Easy launch mechanisms Seamless Parallel and Remote Debugging Powerful Process Control Features MPI Message Queue Display High Degree of Scalability Scriptability for unattended operation

Preparing for debugging • Compiling Your Application • Provide Total. View with debug symbols • Add '-g' to your compile line • Turn off optimizations • Remove any optimization flags ‘-o’ 69

Starting an MPI Job Within Total. View Indirect launch • Choose MPI implementation • Set parameters • Enable Memory Debugging • Indicate your MPI Start from command line 70 • mpirun -tv -np 4 my_program (mpich) • totalview poe -a -np 4 my_program

Running Total. View with Si. Cortex Applications • The Total. View Debugger runs as a cross-debugger within the Si. Cortex-MIPS Linux environment. • The Si. Cortex version is a 64 -bit application runs on a x 86 - 64 system running a 64 -bit kernel. • Debugging on Si. Cortex uses the remote features of Total. View 71

Running Total. View with Si. Cortex Applications • TVD needs to execute a command on the target system from the development host. • By default, this version uses the ssh -x command. • It is suggested to use ssh set so that allows password-less commands. • The program’s executable file must be visible from both the development host and the target system. • Place the executables in a directory that is visible on both machines through the same path. • Having the executable visible in separate directories that are accessed through the same path on both machines will also work. 72

Running Total. View with Si. Cortex Applications The Si. Cortex version of Total. View uses a different set of naming conventions, using an ”sc” prefix • sctv 8 instead of tv 8 • sctototalview instead of totalview • sctv 8 cli vs. tv 8 cli for the Command Line I/F 73

Running Total. View with Si. Cortex Applications TVD must debug the MIPS version of srun, not the x 86 -64 version of srun. Total. View can be Invoked as follows: sctv 8 -r Si. Cortex_node. /srun -a srun_arguments Via the GUI: • Use the File > New Program dialog box. • Within the Parallel tab, select Si. Cortex from the pull- down list. (This is the preferred way to start MPI programs from within TVD. ) 74

Root Window with MPI • Status Info • T = stopped • B = Breakpoint • E = Error • W = Watchpoint • R = Running • M = Mixed • Navigation • Dive to refocus • Dive anew to get a second process window 75

Call Graph • Quick view of program state • Each call stack is a path • Functions are nodes • Calls are edges • Labeld with the MPI rank • Construct process groups • Look for outliers 76

Process Control Concepts • Each process window is always focused on a specific process. • Process focus can be easily switched • Processes can be ‘held’ - they will not run till unheld. • Breakpoints can be set to stop the process or the group • Breakpoint and command scope can be simply controlled 77

Switching Processes • • You can switch the focus of the process window by using the P+ and P- buttons on the toolbar. – P+ takes you to the next process. – P- takes you to the previous process. You can also navigate directly to processes by diving on process entries in the root window. – The next slide describes using the root window in more detail.

Process Control with MPI • Process control commands have a 'scope’. For MPI debugging the following scopes are interesting: • • Arbitrary sets of processes can be controlled by defining a process group with the command line interface. • • Control Group Scope: the entire MPI job, including starter (if there is a separate starter) Share Group Scope: all the rank processes (if you are focused on a rank process) Process Scope: the process that the Process Window is focused on. See later slides and Chapter 11 of the users manual Nothing MPI specific about process control

Looking at Variables across Processes • • • Total. View allows you to look at the value of a variable in all MPI processes • Right Click on the variable • Select the View > View Across Total. View creates an array indexed by process You can filter and visualize

View MPI Message Queues • Information visible whenever MPI rank processes are halted • Provides information from the MPI layer • Unexpected messages • Pending Sends • Pending Receives • Use this info to debug • Deadlock situations • Load balancing 81

Message Queue Graph • Hangs & Deadlocks • Pending Messages • Receives • Sends • Unexpected • Inspect • Individual entries • Patterns 82

Message Queue Debugging • Filtering • Tags • MPI Communicators • Cycle detection • Find deadlocks 83

Large Jobs: Subset Attach Total. View does not need to be attached to the entire job • You can be attached to different subsets at different times through the run • You can attach to a subset, run till you see trouble and then 'fan out' to look at more processes if necessary. • This greatly reduces overhead 84

Large Jobs: Strategies • Reduce N • Problem: Each process added requires overhead • Strategy: Reduce the number of processes Total. View is attached to • Simply reducing N is best, however data or algorithm may require large N • Technique: subset attach mechanism • Focus Effort • Problem: Some debugger operations are much more intensive than others, when multiplied by N this is a big deal • Strategy: Reduce the interaction between the debugger and the processes • Technique: Use Total. View's process control features to • Avoid single stepping • Focus on one or a small set of processes 85

Large Jobs: Focus on One Process • If you want to single step through a section of code • Perhaps do it on one process only… • Set a process-width breakpoint at the beginning • Once everything you want/expect is lined up, hold one process • Do a group-width ‘go’ to get all the other processes running • Unhold that one process and change the width of the commands in the tool bar to process • Next and step that one process • The other processes are running so they will participate in communication with your process of interest 86

Thanks! QUESTIONS? 87

88