Parallel Computing Laxmikant Kale http charm cs uiuc

Parallel Computing Laxmikant Kale http: //charm. cs. uiuc. edu

Advent of parallel computing • “Parallel computing is necessary to increase speeds” – cry of the ‘ 70 s – processors kept pace with Moore’s law: • Doubling speeds every 18 months • Now, finally, the time is ripe – uniprocessors are commodities (and proc. speeds shows signs of slowing down) – Highly economical to build parallel machines 2

What is Parallel Computing? • Use of multiple processors to solve a single computational problem faster. – Distinct from distributed computing • Why parallel computing? – “Parallel computing is necessary to increase speeds” – cry of the ‘ 70 s – processors kept pace with Moore’s law: • Doubling speeds every 18 months • Now, finally, the time is ripe – uniprocessors are commodities (and proc. speeds shows signs of slowing down) – Highly economical to build parallel machines 3

Why parallel computing • It is the only way to increase speed beyond uniprocessors – Except, of course, waiting for uniprocessors to become faster! – Several applications require orders of magnitude higher performance than feasible on uniprocessors • Cost effectiveness: – older argument – in 1985, a supercomputer cost 2000 times more than a desktop, yet performed only 400 times faster. – So: combine microcomputers to get speed at lower costs – Incremental scalability: • can get inbetween performance points with 20, 50, 100, … processors – But: • You may get speedup lower than 400 on 2000 processors! • Microcomputers became faster, killing supercomputers, effectively 4

Technology Trends The natural building block for multiprocessors is now also about the fastest! 5

Architectural Trends • Greatest trend in VLSI generation is increase in parallelism – Up to 1985: bit level parallelism: 4 -bit -> 8 bit -> 16 -bit • slows after 32 bit • adoption of 64 -bit now under way, 128 -bit far (not performance issue) • great inflection point when 32 -bit micro and cache fit on a chip – Mid 80 s to mid 90 s: instruction level parallelism • pipelining and simple instruction sets, + compiler advances (RISC) • on-chip caches and functional units => superscalar execution • greater sophistication: out of order execution, speculation, prediction – to deal with control transfer and latency problems 6

Economics • Commodity microprocessors not only fast but CHEAP • Development cost is tens of millions of dollars (5 -100 typical) • BUT, many more are sold compared to supercomputers – Crucial to take advantage of the investment, and use the commodity building block – Exotic parallel architectures no more than special-purpose • Multiprocessors being pushed by software vendors (e. g. database) as well as hardware vendors • Standardization by Intel makes small, bus-based SMPs commodity • Desktop: few smaller processors versus one larger one? – Multiprocessor on a chip 7

What to Expect? • Parallel Machine classes: – Cost and usage defines a class! Architecture of a class may change. – Desktops, Engineering workstations, database/web servers, suprtcomputers, • Commodity (home/office) desktop: – less than $5, 000 – possible to provide 5 -25 processors for that price! – Driver applications: • games, video /signal processing, • possibly “peripheral” AI: speech recognition, natural language understanding (? ), smart spaces and agents • New applications? 8

Engineeering workstations • Price: less than $100, 000 (used to be): – new proce level acceptable may be $50, 000 – 100+ processors, large memory, – Driver applications: • • CAD (Computer aided design) of various sorts VLSI Structural and mechanical simulations… Etc. (many specialized applications) 9

Commercial Servers • Price range: variable ($10, 000 - several hundreds of thousands) – defining characteristic: usage – Database servers, decision support (MIS), web servers, e-commerce • High availability, fault tolerance are main criteria • Trends to watch out for: – Likely emergence of specialized architectures/systems • E. g. Oracle’s “No Native OS” approach • Currently dominated by database servers, and TPC benchmarks – TPC: transactions per second – But this may change to data mining and application servers, with corresponding impact on architecure. 10

Supercomputers • “Definition”: expensive system? ! – Used to be defined by architecture (vector processors, . . ) – More than a million US dollars? – Thousands of processors • Driving applications – – – – Grand challenges in science and engineering: Global weather modeling and forecast Rational Drug design / molecular simulations Processing of genetic (genome) information Rocket simulation Airplane design (wings and fluid flow. . ) Operations research? ? Not recognized yet Other non-traditional applications? 11

Scientific Computing Demand 12

Engineering Computing Demand • Large parallel machines a mainstay in many industries – Petroleum (reservoir analysis) – Automotive (crash simulation, drag analysis, combustion efficiency), – Aeronautics (airflow analysis, engine efficiency, structural mechanics, electromagnetism), – Computer-aided design – Pharmaceuticals (molecular modeling) – Visualization • in all of the above • entertainment (films like Toy Story) • architecture (walk-throughs and rendering) – Financial modeling (yield and derivative analysis) – etc. 13

What is Challenging? • Writing parallel programs is difficult – Office worker analogy • Issues of • • Coordination: I thought you were going to get the pizza Asynchrony: what happens before what Race conditions: can’t determine which will happen first And finally: Performance! 14