PAPI 3 0 8 1 on Blue Gene

PAPI 3. 0. 8. 1 on Blue Gene L Using network performance counters to layout tasks for improved performance

Presentation overview n n Project objectives PAPI explanation Blue Gene L explanation Current state of research

Project objectives n Upgrade PAPI on BG/L n n n Provide interface for network counters Allow Lawrence Livermore National Lab users to also have access to PAPI Using network counters to place tasks optimally on BG/L

PAPI – Intro Courtesy of http: //icl. cs. utk. edu/papi/

PAPI – Intro n n PAPI useful to profile your own programs. Many tools based on PAPI n n Papi. Ex – Command line measurement tool Perf. Suite – Aggregate measurement and statistical profiling package and API HPCToolkit – Statistical profiling package Many more!

PAPI – Supported platforms n n n n IBM – POWER 3, 604 e, POWER 4 Cray T 3 E, Cray X 1 AMD – Athlon, Opteron Intel – P 1 to P 4, Itanium I and II Ultra. Sparc I, II & III MIPS R 10 K, R 12 K, R 14 K Alpha

PAPI – Generic Interface n Call sequence for generic interface n n n PAPI_library_init – Initialize memory for PAPI’s data structures PAPI_create_eventset – Create an empty list of events PAPI_add_event – Add events to be counted PAPI_start – Begin counting all events within the specified eventset PAPI_stop – Stop all counters and read their current values

PAPI – Events: Presets n Presets – list of predefined events implemented on all systems where they can be supported n n Not all presets available on every architecture (e. g. BG/L has no cache lower than L 3 – thus L 1 cache hit preset not applicable) Native events form the basic building blocks for PAPI presets

PAPI – Events: Presets Courtesy of http: //icl. cs. utk. edu/papi/

PAPI – Events: Native n n In addition to the predefined PAPI preset events, the PAPI library also exposes a majority of the events native to each platform Can be added to eventsets in the same manner as presets

PAPI – Events: Native

PAPI – Internals n Array of eventsets is the main portion

PAPI – Other features n n n Multiplexing – If there are not enough hardware counters Thread safe – Profiling is thread safe Overflow detection – Hardware counters have limited space

PAPI – PAPI 2 vs PAPI 3 n PAPI 3 significantly reduced overheads for starting, stopping and reading the counters Courtesy of http: //icl. cs. utk. edu/papi/

PAPI – PAPI 2 vs PAPI 3 n n Better native event support in PAPI 3 Better thread support in PAPI 3 Overflow and Profiling enhancements in PAPI 3 Myriad bug fixes and code cleanup in PAPI 3

PAPI – PAPI 2 vs PAPI 3 n n Overlapping eventsets supported in PAPI 2 Minor changes in the API – mostly dereferencing variables

Blue Gene L – Intro n n 65, 536 nodes connected in 64 x 32 3 D torus Nodes made up of Power. PC 440 embedded processors Smaller than most super computers Consumes less power

Blue Gene L

Blue Gene L - Networks 3 D torus network (node to node) n Tree network (broadcasts) n

Blue Gene L – HW counters n n n 48 universal performance counters 4 floating point unit counters Counters 32 bit – must use virtual counters to prevent overflow

Blue Gene L – HW counters

Research – Overall goals n n Network hardware counters new Use network counters to determine traffic between tasks Try to optimize placement of tasks to minimize communication latency Given counts and distances: cost = counts * distance. Minimize over all nodes

Research – Counting n First goal to determine what is being counted

Research – Networks n For each MPI call – determine which network counters are being used n n n Tree is supposed to be for broadcasts Torus is supposed to be for point to point communication Ambiguities in the specification

Research – Future decisions n How to profile a target application n Manually insert PAPI instrumentation: a lot of work Instrument binaries with counting code What information to store n n All counts on each node: a lot of data Sample of all nodes: not as accurate (what if the tasks behave / communicate differently?

Research – Future decisions n How to use collected information n n Profile an application to obtain counter feedback to determine optimized static task layout Dynamically migrate tasks in response to counters