TAU Performance System Performance Tools BOF SC 07

TAU Performance System Performance Tools BOF, SC’ 07 5: 30 pm – 7 pm, Tuesday, A 9 Sameer S. Shende sameer@cs. uoregon. edu http: //tau. uoregon. edu Performance Research Laboratory University of Oregon

Acknowledgements r r r r r Dr. Allen D. Malony, Professor Alan Morris, Senior software engineer Wyatt Spear, Software engineer Scott Biersdorff, Software engineer Matt Sottile, Research faculty Rob Yelle, Research faculty Kevin Huck, Ph. D. student Aroon Nataraj, Ph. D. student Shangkar Myangalam, Ph. D. student Brad Davidson, Systems administrator SC’ 07: Performance Tools BOF TAU Performance System 2

TAU Parallel Performance System r http: //tau. uoregon. edu/ r Multi-level performance instrumentation ¦ r r Multi-language automatic source instrumentation Flexible and configurable performance measurement Widely-ported parallel performance profiling system Computer system architectures and operating systems ¦ Different programming languages and compilers ¦ r Support for multiple parallel programming paradigms ¦ Multi-threading, message passing, mixed-mode, hybrid SC’ 07: Performance Tools BOF TAU Performance System 3

What is TAU? r r r Portable, profiling and tracing toolkit BSD style license Automatic source level instrumentation (PDT, Opari [FZJ]) ¦ ¦ r Routine, and loop level Instrumentation optimization (TAU_THROTTLE) Measurement ¦ ¦ Callpath, phase, parameter based profiling PAPI [UTK] support in profiling and tracing Ø r Analysis tools ¦ ¦ ¦ r One or more native, preset events Parallel profile analysis (Para. Prof) Performance data management (Perf. DMF database) Performance data mining (Perf. Explorer) Kernel monitoring and KTAU SC’ 07: Performance Tools BOF TAU Performance System 4

What is new in TAU and PDT? r r TAU v 2. 17 and PDT v 3. 12 released at SC’ 07 [tau. uoregon. edu] Support for new platforms ¦ ¦ IBM BG/P (Linux) Si. Cortex Cray XT (Compute Node Linux) Windows Cluster 2003 r Improved support for Vampir. Trace [TU Dresden] for atomic events, native OTF generation r Automatic wrapper library generation (tau_wrap) for HDF 5, I/O… Enhanced Eclipse/PTP plugin for tool interoperability Perf. Explorer: Custom charts, multiple database support Para. Prof supports multiple profile formats, databases r r r ¦ ¦ r PERIXML, TAU, perfsuite, cube 2 & 3, mpi. P, HPMtoolkit, gprof… Support for storing metadata in profiles, TAU portal, Perf. DMF PDT now supports type information in PDB files GFortran parser SC’ 07: Performance Tools BOF TAU Performance System 5

TAU Demos at SC’ 07 r ASC/NNSA Booth #1617, demo station #2 ¦ ¦ ¦ r r Si. Cortex Booth ANL Booth (KTAU presentation) ¦ r Wednesday (11/14): 12 -1 pm, 3 -5 pm Thursday (11/15): 10 -11 am Schedule available at tau. uoregon. edu Thu. 12 -1 pm Paper: ¦ “Ghost in the machine: Observing the Effects of Kernel Operation in Parallel Application Performance” A. Nataraj, A. Morris, A. Malony, M. Sottile, P. Beckman, SC’ 07 A 2/A 5 Wed. 10: 30 am SC’ 07: Performance Tools BOF TAU Performance System 6

Future Research Directions r Improving tool interoperability ¦ ¦ ¦ r Hybrid sampling, instrumentation based measurements ¦ r r OTF [TU Dresden] Total. View [Totalview Tech] Scalasca/KOJAK instrumentation [FZJ] Perf. Suite [NCSA] Kernel measurements for tracking I/O using KTAU and Zepto. OS [ANL] Binary rewriter integration using Dyninst. API [U. Maryland, U. Wisconsin] Improvements in Si. Cortex integrated tool environment TAU Portal, regression testing SC’ 07: Performance Tools BOF TAU Performance System 7

TAU Performance System Architecture SC’ 07: Performance Tools BOF TAU Performance System 8

TAU Performance System Architecture SC’ 07: Performance Tools BOF TAU Performance System 9

Program Database Toolkit (PDT) Application / Library C / C++ parser Fortran parser F 77/90/95 IL IL C / C++ IL analyzer Program Database Files Fortran IL analyzer DUCTAPE SC’ 07: Performance Tools BOF PDBhtml Program documentation SILOON Application component glue CHASM C++ / F 90/95 interoperability TAU_instr Automatic source instrumentation TAU Performance System 10

Building Bridges to Other Tools: TAU SC’ 07: Performance Tools BOF TAU Performance System 11

Using TAU: A brief Introduction r To instrument source code using PDT Choose an appropriate TAU stub makefile in <arch>/lib: % setenv TAU_MAKEFILE /usr/tau-2. 17/x 86_64/lib/Makefile. tau-mpi-pdt-pgi % setenv TAU_OPTIONS ‘-opt. Verbose …’ (see tau_compiler. sh) And use tau_f 90. sh, tau_cxx. sh or tau_cc. sh as Fortran, C++ or C compilers: % mpif 90 foo. f 90 changes to % tau_f 90. sh foo. f 90 ¦ r Execute application and analyze performance data: % pprof (for text based profile display) % paraprof (for GUI) SC’ 07: Performance Tools BOF TAU Performance System 12

TAU’s Para. Profile Browser: Manager Multiple Perf. DMF databases SC’ 07: Performance Tools BOF TAU Performance System Application Metadata 13

TAU’s Para. Prof Scalable Profile Browser S 3 D: 6400 cores on XT 3+XT 4 System (Jaguar) r Gap represents XT 3 nodes SC’ 07: Performance Tools BOF TAU Performance System 14

S 3 D Scatter Plot: Visualizing Hybrid XT 3+XT 4 6400 cores r Red nodes are XT 4, blue are XT 3 SC’ 07: Performance Tools BOF TAU Performance System 15

Perf. DMF Architecture K. Huck, A. Malony, R. Bell, A. Morris, “Design and Implementation of a Parallel Performance Data Management Framework, ” ICPP 2005. SC’ 07: Performance Tools BOF TAU Performance System 16

TAU Portal: Launch Para. Prof/Perf. Explorer SC’ 07: Performance Tools BOF TAU Performance System 17

Perf. Explorer: Regression Testing SC’ 07: Performance Tools BOF TAU Performance System 18

Perf. Explorer: S 3 D Total Runtime Breakdown WRITE_ SAVEFILE MPI_Wait 12, 000 cores! SC’ 07: Performance Tools BOF TAU Performance System 19

TAU in Eclipse/PTP Environment SC’ 07: Performance Tools BOF TAU Performance System 20

Support Acknowledgements US Department of Energy (DOE) ¦ Office of Science Ø MICS, Argonne National Lab ¦ ASC/NNSA Ø University of Utah ASC/NNSA Level 1 Ø ASC/NNSA, Lawrence Livermore National Lab r US Department of Defense (Do. D) r NSF HEC-RTF and SDCI r Research Centre Juelich r TU Dresden r Los Alamos National Laboratory r Para. Tools, Inc. r PSC, NCSA, and U. Oregon r SC’ 07: Performance Tools BOF TAU Performance System 21