A Strategy for the Future of High Performance

A Strategy for the Future of High Performance Computing? Advanced Computing Laboratory Los Alamos National Laboratory Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory

Observations: The US Supercomputing Industry l l l All US high-performance vendors are building clusters of SMPs (with the exception of Tera) Each company, IBM, SGI, Compaq, HP, and SUN has a different version of Unix Each company attempts to scale system software designed for database, internet, technical and servers This fractured market forces 5 different parallel file systems, fast message implementations, etc Supercomputer companies tend to go out of business Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 2

New Limitations People used to say: “The number of Tflops available is limited only by the amount of money you wish to spend” The Reality: We are at a point where our ability to build machines from components exceeds our ability to admin, program and run them But we do it anyway. Many large clusters are being installed. . . Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 3

Scalable System Software is currently the weak link Software for Tflop clusters of SMPs is hard l l l System administration, configuration, booting, management, & monitoring Scalable smart NIC messaging (zero copy) Cluster/Global/Parallel File System Job queuing and running I/O (scratch, prefetch, NASD) Fault tolerance and on-the-fly reconfiguration Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 4

Why use Linux for clusters of SMPs, and as a basis for system software research? l l The OS for scalable clusters needs more research Open Source! (it’s more then just geek chic) No lawyers, no NDAs, no worries mate! n Visible code improves faster n The whole environment, or just the mods can be distributed n Scientific collaboration is just an URL away. . . n l Small, well designed, stable, mature, kernel ~240 K lines of code without device drivers n /proc filesystem and dynamically loadable modules n n The OS is extendable, optimizable, tunable Linux is a lot of fun (Shagadelic, Baby!) Did I mention no lawyers? Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 5

Isn’t Open Source hype? Do you really need it? A very quick example Supermon and Superview: High Performance Cluster Monitoring Tools Ron Minnich, Karen Reid, Matt Sottile Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 6

The problem: get really fast stats from a very large cluster Monitor hundreds of nodes at rates up to 100 Hz l Monitor at 10 Hz without significant impact on the application l Monitor hardware performance counters l Collect a wide range of kernel information (disk blocks, memory, interrupts, etc) l Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 7

Solution Modify the kernel so all the parameters can be grabbed without going though /proc l Tightly coupled clusters can get realtime monitoring stats. l This is not of general use to the desktop, and web server markets l Stats for 100 nodes takes about 20 ms l Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 8

Superview: the Java tool for Supermon Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 9

Scalable Linux System Software Where should we concentrate our efforts? Some areas for improvement…. Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 10

Software: The hard part Linux environments (page 1) l Compilers n n n l F 90 (PGI, Absoft, Compaq) F 77 (GNU, PGI, Absoft, Compaq, Fujitsu) HPF (PGI, Compaq? ) C/C++ (PGI, KAI, GNU, Compaq, Fujitsu) Open. MP (PGI) Metrowerks Code Warrior for C, C++, (Fortran? ) Debuggers Totalview… maybe, real soon now, almost? n gdb, DDD, etc. n Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 11

Software: The hard part Linux environments (page 2) l Message Passing MPICH, PVM, MPI MSTI, Nexus n OS Bypass: n – ST, FM, AM, PM, GM, VIA, Portals, etc – Fast Interconnects: Myrinet, Gig. E, Hi. PPI, SCI l Shared Memory Programming n l Pthreads, Tulip-Threads, etc. Parallel Performance Tools n TAU, Vampir, PGI PGProf, Jumpshot, etc Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 12

Software: The hard part Linux environments (page 3) l File Systems & I/O n e 2 fs (native), NFS n PVFS, Coda, GFS n MPI-IO, ROMIO l Archival Storage n HPSS l & ADSM clients Job Control n LSF, Pete Beckman PBS, Maui Advanced Computing Laboratory Los Alamos National Laboratory 13

Software: The hard part Linux environments (page 4) l Libraries and Frameworks n BLAS, OVERTURE, POOMA, Atlas n Alpha math libraries (Compaq) l System Administration n Building and booting tools n Cfengine n Monitoring and management tools n Configuration database n SGI Project Accounting Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 14

Software for Linux clusters A report card (current status) Compilers ………………. . A Parallel debuggers ……………. . . I Message passing ……………. …. …. . AShared memory prog. …………. . A Parallel performance tools …………. . C+ File Systems …………………. . D Archival Storage ………………. ……. . C Job Control ………………. . BMath Libraries …………………. . B Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 15

Summary of the most important areas l First Priority n n n Cluster management, administration, images, monitoring, etc Cluster/parallel/global file systems Continued work on scalable messaging Faster, more scalable SMP Virtual memory optimized for HPC TCP/IP improvements Wish List NIC boot, BIOS NVRAM, Serial console OS bypass standards in the kernel Tightly-coupled scheduling, accounting Newest Drivers Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 16

Honest cluster costs: publish the numbers How many sysadmins and programmers are we required for support? l What are the service and replacement costs? l How much was hardware integration? l How many users can you support and at what levels? l How much was the hardware? l Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 17

Tera-Scale SMP Cluster Architecture Network Attached Secure Disks Gigabit Multistage Interconnection Fabric Compute Nodes Control Node Gigabit Ethernet Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory Control Node Unit 18

Advanced Computing Laboratory Los Alamos National Laboratory

Advanced Computing Laboratory Los Alamos National Laboratory

Let someone else put it together Compaq l Dell l Penguin Computing l Alta Tech l VA Linux l DCG l Paralogic l Microway Ask about support l Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 21

Cluster Benchmarking Lies, Damn Lies, and the Top 500 Vendor Published Linpack, Latency, and Bandwidth numbers are worthless l l l Make MPI zero-byte messaging a special case (improves latency numbers) Convert multiply flops to addition, recount flops Hire a Linpack consultant to help you achieve “the number” the vendor promised “We unloaded the trucks, and 24 hrs later, we calculated the size of the galaxy in acres. ” For $15 K and 3 rolls of duct tape I built a supercomputer in my cubicle…. Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 22

Plug-in Framework for Cluster Benchmarks Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 23

MPI Message Matching Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 24

Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 25

Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 26

Conclusions l l l Lots of Linux clusters will be at SC 99 The Big 5 vendors do not have the critical mass to develop the system software for multi-teraflop clusters The HPC community (labs, vendors, universities, etc. ) needs to work together The hardware consolidation is nearly over, the software consolidation is on its way A Linux-based “commodity” Open Source strategy could provide a mechanism for: open vendor collaboration n academic and laboratory participation n one Open Source software environment n Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 27

News and Announcements: The next Extreme Linux conference will be in Williamsburg in October. The call for papers will be out soon, start preparing those technical papers… l There will be several cluster tutorials at SC 99. Remy Evard, Bill Saphir, and Pete Beckman will be running one focused on system administration and user environment for large clusters. l Pete Beckman Advanced Computing Laboratory Los Alamos National Laboratory 28