MPI and OFA Divergent interests Dan Caldwell VP

MPI and OFA Divergent interests? Dan Caldwell, VP WW Channel Sales Scali, Inc

The role of OFA in HPC § The traditional role of Open Fabrics focuses on cables that connect motherboards, switches, and storage in clusters § Growth of processors in HPC expected to be 30% in 2010 (per IDC), and number of cores per processor is 55% (estimate) and increasing § Therefore the bulk of the “HPC Interconnect Fabric” is moving from cables to the server motherboard and the various local buses 2 April 2008 OFA Presentation

MPI Performance in HPC systems § Scali has always been the performance MPI leader – http: //www. supercomputingonline. com/article. php? sid= 15357 § However, we suggest that OFA adopt a recommended performance measurement methodology – Productivity, or ‘jobs per day’, on a cluster – How do you know how well OFA is doing? § OFA must examine the performance within a multi-core node – A 16 core “Personal Supercomputer” will still run legacy MPI applications 3 April 2008 OFA Presentation

Single Node, 8 cores, osu_bw, 8 byte, Intel Xeon 3. 00 GHz (X 5365) 4 April 2008 OFA Presentation

MPI enables other HPC functionality § Infiniband Trunking – Combining IB channels for greater throughput – Done for Sun / Tsukuba University in Japan § Suspend / Resume and Checkpoint – Restart – Generic functionality - not application specific – Demonstrated job migration with HPC 4 U in Brussels, February 11, 2008 § And…. MPI based Power Management – Today – AMD / Barcelona only 5 April 2008 OFA Presentation

”Waiting as fast as it can” (CPU spinning) – examples (AMD Barcelona Quad Core): Power Saving Potential Zone Considering 20 microseconds to change power states in an Quad core Opteron, latency above 10 milliseconds in an MPI collective can trigger a ”throttle down”. In a call with a 10 millisecond latency, 20 microseconds to throttle down plus 200 to re-establish full speed would use only 2. 5% of the time of the call. 6 April 2008 OFA Presentation

Real World Initial Tests - SPEC MPI 2007 7 April 2008 OFA Presentation

Conclusions § MPI and OFA need to co-exist, but we both need to expand our reach in HPC § Open Fabrics needs to embrace new functionality, performance metrics, and definitions of ‘fabric’ § Power savings, user-model (core affinity policies), best practices in interconnect design, and emphasis on continued performance across the entire HPC system is critical to the relevancy of OFA. 8 April 2008 OFA Presentation