Architecture or Parallel Computers CSC ECE 506 Summer

Architecture or Parallel Computers CSC / ECE 506 Summer 2006 Introduction / Overview 5/22/2006 Dr Steve Hunter

Architecture of Parallel Computers • Taught jointly by Dr Ed Gehringer and Dr Steve Hunter • Course days – Monday – Wednesday 4: 00 – 6: 45 4: 00 – 5: 15 • Goal: Understand the interaction of hardware and software with respect to parallel systems design and implementation. • Textbook “Parallel Computer Architecture”, by Culler and Singh • Selected papers possible Arch of Parallel Computers CSC / ECE 506 2

Architecture of Parallel Computers • Steve’s Info: – – NCSU Adjunct Professor IBM Corporation Website: http: //www. ee. duke. edu/~shunter/ email: hunters@us. ibm. com • Academic – Auburn University – NC State University – Duke University BSEE MSEE Ph. D • IBM Corporation – IBM Networking Division – Systems and Technology Group 14 years 8 years • Areas of Interest – Systems and Network Architecture and Technology – Computer and Network Performance and Dependability – Server Clustering and Software Dependability Arch of Parallel Computers CSC / ECE 506 3

Course Outline (Tentative) Mon. 4: 00 Mon. 5: 30 Wed. 4: 00 5/225/24 1. Overview of parallel computation (1) 2. Message-passing and data-parallel models (3 -5) 3. Steps in parallelization (6) 5/295/31 Memorial Day 4. Parallelizing the Ocean application (7) 6/56/7 5. Data-parallel algorithms (8) 6. Cache organization (10) 7. The cache-coherence problem/interleaved memory (12) 6/126/14 8. Invalidation-based cache-coherence protocols (13) 9. Update-based coherence protocols and perf. (14) Test 1 6/196/21 10. Scalable multiprocessors (16) 11. Realizing programming models in scalable systems (17) 12. Design space for communication architectures 6/266/'28 13. Scalable cache coherence (19) 14. Directory-protocol correctness and performance (20) 7/'37/5 Independence Day 15. The Silicon Graphics S 2 MP architecture (21) 7/107/12 16. Extending cache coherence (15) 17. Open MPI Test 2 7/177/19 18. Open RDMA, open fabrics 19. Memory consistency (22) 20. Relaxed memory-consistency models (23) 7/247/26 21. Interconnection network topologies (25) 22. Routing in interconnection networks (26) 23. Switch design (27) 7/318/2 8/78/9 Final exam http: //courses. ncsu. edu/csc 506/lec/052/lectures/syllabus. html Arch of Parallel Computers CSC / ECE 506 4

What is Parallel Computer Architecture? • A Parallel Computer is a collection of processing elements that cooperate to solve large problems fast • Some broad issues: – Resource Allocation: » how large a collection? » how powerful are the elements? » how much memory? – Data access, Communication and Synchronization » how do the elements cooperate and communicate? » how are data transmitted between processors? » what are the abstractions and primitives for cooperation? – Performance and Scalability » how does it all translate into performance? » how does it scale? Arch of Parallel Computers CSC / ECE 506 5

Historical Perspective • Parallel computing was represented by competing models and corresponding unique architectures, no clear path for growth • Competing Methods – – – Dataflow Systolic Arrays SIMD (bit serial) Shared Memory Message passing • Confusion occurs over which model to use paralyzed parallel software development – Section 1. 2 shows several architectures. – Shared-Memory Multiprocessors » Bus-based; Crossbar-based; * MIN-based – Message Passing Machines (Hypercube) » IBM SP 2 Architecture Arch of Parallel Computers CSC / ECE 506 6

Why Study Parallel Computer Architecture? • Role of a computer architect: – To design and engineer the various levels of a computer system to maximize performance and programmability within limits of technology and cost. • Parallelism: – – Provides alternative to faster clock for performance Applies at all levels of system design Is a fascinating perspective from which to view architecture Traditionally central in information processing elements in the same locality – However, greater networking bandwidth is expanding parallelism over greater distances. Arch of Parallel Computers CSC / ECE 506 7

Parallel Computation: Why and Why Not? • Pros – – Performance Cost-effectiveness (commodity parts) Smooth upgrade path Fault Tolerance • Cons – Difficult to parallelize applications – Requires automatic parallelization or parallel program development – Software! Arch of Parallel Computers CSC / ECE 506 8

Is Parallel Computing Inevitable? • Application demands: (the need for computing cycles) – Petroleum (reservoir analysis) – Automotive (crash simulation, drag analysis, combustion efficiency) – Aeronautics (airflow analysis, engine efficiency, structural mechanics, electromagnetism) – Computer-aided design – Pharmaceuticals (molecular modeling) – Visualization » in all of the above » entertainment (films like Toy Story, The Hulk) » architecture (walk-throughs and rendering) – Financial modeling (yield and derivative analysis) – Search Engines – etc. Arch of Parallel Computers CSC / ECE 506 9

Application Trends • Application demand for performance fuels advances in hardware, which enables new applications, which. . . – Cycle drives exponential increase in microprocessor performance – Drives parallel architecture harder » most demanding applications New Applications More Performance • Range of performance demands – Need range of system performance with progressively increasing cost Arch of Parallel Computers CSC / ECE 506 10

Speedup Performance (p processors) • Speedup (p processors) = Performance (1 processor) • For a fixed problem size (input data set), performance = 1/time • Speedup fixed problem (p processors) = Arch of Parallel Computers CSC / ECE 506 Time (1 processor) Time (p processors) 11

Is Parallel Computing Inevitable? • Technology Trends – – Chip technology continues to increase in density Driving frequency of single core designs requires too much power Use of commodity or off-the-shelf technology for low costs Multi-core processing becoming common among mainstream microprocessors (e. g. , AMD, IBM, Intel) – Greater interconnect bandwidth becoming generally available » Standard interconnects: Infiniband, 10 Gb Ethernet • Architecture Trends – Packaging parallel solutions in a common chassis » e. g. , Blade servers (IBM, HP, Dell, etc. ) – Software being packaged for mainstream solutions » e. g. , Windows Compute Cluster Server 2003 – High availability commonly achieved by clustering of processing elements Arch of Parallel Computers CSC / ECE 506 12

Is Parallel Computing Inevitable? • Economics – The reducing costs of low end servers (dual and quad socket) with high bandwidth of interconnects is driving applications to be parallel – Commodity microprocessors not only fast but CHEAP » Development costs tens of millions of dollars » BUT, many more are sold compared to supercomputers » Crucial to take advantage of the investment, and use the commodity building block – Multiprocessors being pushed by software vendors (e. g. database) as well as hardware vendors – Standardization makes small, bus-based SMPs commodity – Desktop: few smaller processors versus one larger one? – Multiprocessor on a chip? Arch of Parallel Computers CSC / ECE 506 13

Scale Up vs Scale Out Model Scale Up / SMP Computing Large SMP Large Parallel Clusters x 455 High Density Rack Mount x 445 Blade. Center™ x. Series 335/e. Server 325 Scale Out / Distributed Computing Arch of Parallel Computers CSC / ECE 506 14

Blade Server Example - Blade. Center Nov 2002 Blade. Center 7 U Chassis Form Factor Highest Density, Lowest cost Super power efficient, Consolidated Management March 2004 Blade. Center T 8 U Chassis Form Factor Highly rugged, Telco AC/DC, Long Life, NEBS, Air Filtration Jan 2006 Blade. Center H 9 U Chassis Form Factor Ultra High Performance 4 x. IB/10 Gb Backplane New Management Module Compatible Set of Blades and Switches Web hosting/serving FSS, File/Print Geophysical Analysis Collaboration Graphic Rendering Telco/Core Applications Government Military Rugged Industrial DC Medical HPC Applications Technical Clusters Virtual Enterprise Solutions Future I/O One family, many applications, many environments, long term investment protection. Blade. Center Simply Smarter IT Arch of Parallel Computers CSC / ECE 506 15

Blade Server Example – Blade. Center H • Fourteen Blades in a 9 U Chassis Form Factor – Blade and switch compatibility across Blade. Center and Blade. Center-T • High performance networking fabrics – New high performance switches and blade I/O – Corresponding bridge bays for protocol translation • Power Enhancements – Four front load 2900 W Power Supplies Arch of Parallel Computers CSC / ECE 506 16

Blade. Center Overview • Switching Modules – Ethernet – Fiber Channel – Infiniband • Blade I/O Card (or local drive) – I/O card matches switch technology in corresponding slot Arch of Parallel Computers CSC / ECE 506 17

Blade. Center H Architecture I/O Bridge Blade 1 HS Switch 2 Blade 2 I/O Bridge 3/ SM 3 . . . HS Switch 3 High-speed Switch • Ethernet or Infiniband • 4 x (16 wire) blade links • 4 x (16 wire) bridge links • 1 x (4 wire) Mgmt links • Uplinks: Up to 12 x links for IB and at least four 10 Gb links for Ethernet HS Switch 4 I/O Bridge 4 / SM 4 I/O Bridge Switch Module 1 Switch Module 2 Blade 14 Mgmt Mod 1 • e. g. , Ethernet, Fibre Channel, Passthru • Dual 4 x (16 wire) wiring internally to each HSSM Mgmt Mod 2 Arch of Parallel Computers CSC / ECE 506 18

Infini. Band on Blade. Center H Enabling High Performance and Virtualized I/O • Expanding Blade. Center Ecosystem with Cisco Systems – Switch module and daughter card designed for Blade. Center H – Daughter card provides dual port 4 x (10 G) Infini. Band connectivity to each blade • Help Reduce Data Center Complexity – Reduce the number of adapters, cables, and switch ports required – Manage the addition or removal of I/O or storage bandwidth centrally – Enable users to adjust resources on demand without downtime • High Performance Computing Features – Leverages RDMA to deliver low latency performance – Delivers higher bandwidth connectivity (160 Gbps to chassis) – Achieve blade port consolidation through remote I/O – I/O Virtualization via Cisco VFrame Blade. Center H Infini. Band Solution provides highspeed, low latency solutions while lowering TCO Arch of Parallel Computers CSC / ECE 506 19

The End Arch of Parallel Computers CSC / ECE 506 20

Grid Example Inter-Grids Inter -Grids Extra -Grids Cactus Intra-Grids Intra -Grids (SF) Express Project Grid NAS/SAN NTG Grid VPN NAS/SAN 2003 Commerce with Trusted Partners Fin. Services MFG 2006+ "Full Commercialization" with unknown partners Arch of Parallel Computers CSC / ECE 506 21 Courtesy of Ellen Stokes