EECE 571 R Harnessing Massively Parallel Processors http

EECE 571 R -- Harnessing Massively Parallel Processors http: //www. ece. ubc. ca/~matei/EECE 571/ Lecture 1: Introduction Matei Ripeanu matei at ece dot ubc dot ca Acknowledgement: some slides borrowed from presentations by Jim Demel, Horst Simon, Mark D. Hill, David Patterson, Saman Amarasinghe, Richard Brunner, Luddy Harrison, Jack Dongarra 1

Outline all • Why powerful computers must be parallel processors Including your laptops and handhelds • Science problems require powerful computers Other problems too • Why writing (fast) parallel programs is hard But things are improving • Why hybrid architectures (e. g. , GPU based platforms)? • Structure of this course 2

Units of Measure • High Performance Computing (HPC) units are: - Flop: floating point operation - Flops/s: floating point operations per second - Bytes: size of data (a double precision floating point number is 8 bytes) • Typical sizes are millions, billions, trillions… Mega Giga Tera Peta Exa Mflop/s = 106 flop/sec Gflop/s = 109 flop/sec Tflop/s = 1012 flop/sec Pflop/s = 1015 flop/sec Eflop/s = 1018 flop/sec Mbyte = 220 = 1048576 ~ 106 bytes Gbyte = 230 ~ 109 bytes Tbyte = 240 ~ 1012 bytes Pbyte = 250 ~ 1015 bytes Ebyte = 260 ~ 1018 bytes Zetta Zflop/s = 1021 flop/sec Zbyte = 270 ~ 1021 bytes Yotta Yflop/s = 1024 flop/sec Ybyte = 280 ~ 1024 bytes • Current fastest (public) machine ~ 2. 6 Pflop/s - Up-to-date list at www. top 500. org 3

Performance evolution – Top 500 List

all (2007) Why powerful computers are parallel circa 1991 -2006 5

Tunnel Vision by Experts • “I think there is a world market for maybe five computers. ” - Thomas Watson, chairman of IBM, 1943. • “There is no reason for any individual to have a computer in their home” - Ken Olson, president and founder of Digital Equipment Corporation, 1977. • “ 640 K [of memory] ought to be enough for anybody. ” - Bill Gates, chairman of Microsoft, 1981. • “On several recent occasions, I have been asked whether parallel computing will soon be relegated to the trash heap reserved for promising technologies that never quite make it. ” - Kennedy, CRPC Directory, 1994 Slide source: Warfield et al. 6

Technology Trends: Microprocessor Capacity Moore’s Law 2 X transistors/Chip Every 1. 5 years Called “Moore’s Law” Microprocessors have become smaller, denser, and more powerful. Gordon Moore (co-founder of Intel) predicted in 1965 that the transistor density of semiconductor chips would double roughly every 18 months. Slide source: Jack Dongarra 7

Microprocessor Transistors per Chip • Growth in transistors per chip • Increase in clock rate 8

Impact of Device Shrinkage • What happens when the feature size (transistor size) shrinks by a factor of x ? • Clock rate goes up by x because wires are shorter, smaller gate capacities - actually less than x, because of power consumption • Transistors per unit area goes up by x 2 • Die size also tends to increase - typically another factor of ~x • Raw computing power of the chip goes up by ~ x 4 ! - typically x 3 is devoted to either on-chip - parallelism: hidden parallelism such as ILP - locality: caches • So most programs significantly faster, without changing them 9

But there are limiting forces Manufacturing costs and yield problems limit use of density • Cost - Moore’s 2 nd law (Rock’s law): costs go up • Yield - The percentage of the chips that are usable -E. g. , Cell processor (PS 3) is sold with 7 out of 8 “on” to improve yield 10

Power Density Limits Serial Performance 11

Revolution is Happening Now • Chip density is continuing increase ~2 x every 2 years - Clock speed is not! - Number of processor cores may double instead • There is little or no more hidden parallelism (ILP) to be found • Parallelism must be exposed to and managed by software Source: Intel, Microsoft (Sutter) and Stanford (Olukotun, Hammond) 12

Parallelism in 2011? • These arguments are no longer theoretical • All major processor vendors are producing multicore chips - Every machine will soon be a parallel machine - To keep doubling performance, parallelism must double • Which commercial applications can use this parallelism? - Do they have to be rewritten from scratch? • Will all programmers have to be parallel programmers? - New software model needed - Try to hide complexity from most programmers – eventually - In the meantime, need to understand it • Computer industry betting on this change - But does not have all the answers 13

More Limits: How fast can a serial computer be? 1 Tflop/s, 1 Tbyte sequential machine r = 0. 3 mm • Consider the 1 Tflop/s sequential machine: - Data must travel some distance, r, to get from memory to processor. - To get 1 data element per cycle, this means 1012 times per second at the speed of light, c = 3 x 108 m/s. Thus r < c/1012 = 0. 3 mm. • Now put 1 Tbyte of storage in a 0. 3 mm x 0. 3 mm area: - Each bit occupies about 1 square Angstrom, or the size of a small atom. • No choice but parallelism 14

Performance Development 100 Pflop/s 10 Pflop/s SUM 1 Pflop/s 100 Tflop/s N=1 10 Tflop/s 100 Gflop/s 100 Mflop/s N=500 2. 6 PFlops (2010)

Concurrency Levels notebook computer

Moore’s Law reinterpreted • Number of cores per chip will double every two years • Clock speed will not increase (possibly decrease) • Need to deal with systems with large number of concurrent threads - Millions for HPC systems - Thousands for servers - Hundreds for workstations and notebooks • Need to deal with inter-chip parallelism as well as intra-chip parallelism

Outline all • Why powerful computers must be parallel processors Including your laptops and handhelds • Science problems require powerful computers Commercial problems too • Why writing (fast) parallel programs is hard But things are improving • Why hybrid architectures (e. g. , GPU based platforms)? • Structure of the course 18

Computational Science “An important development in sciences is occurring at the intersection of computer science and the sciences that has the potential to have a profound impact on science. It is a leap from the application of computing … to the integration of computer science concepts, tools, and theorems into the very fabric of science. ” -Science 2020 Report, March 2006 Nature, March 23, 2006 19

Drivers for Change • Continued exponential increase in computational power simulation is becoming third pillar of science, complementing theory and experiment • Continued exponential increase in experimental data techniques and technology in data analysis, visualization, analytics, networking, and collaboration tools are becoming essential in all data rich scientific applications 20

Simulation: The Third Pillar of Science • Traditional scientific and engineering method: Theory Experiment (1) Do theory or paper design (2) Perform experiments or build system • Limitations: –Too difficult—build large wind tunnels –Too expensive—build a throw-away passenger jet –Too slow—wait for climate or galactic evolution –Too dangerous—weapons, drug design, climate experimentation Simulation • Computational science and engineering paradigm: (3) Use high performance computer systems to simulate and analyze the phenomenon - Based on known physical laws and efficient numerical methods - Analyze simulation results with computational tools and methods beyond what is used traditionally for experimental data analysis 21

What Supercomputers Do One Example - simulation replacing experiment that is too dangerous 22

Global Climate Modeling Problem • Problem is to compute: f(latitude, longitude, elevation, time) “weather” = (temperature, pressure, humidity, wind velocity) • Approach: - Discretize the domain, e. g. , a measurement point every 10 km - Devise an algorithm to predict weather at time t+dt given t • Uses: - Predict major events, e. g. , El Nino - Use in setting air emissions standards - Evaluate global warming scenarios Source: http: //www. epm. ornl. gov/chammp. html 23

Global Climate Modeling Computation • One piece is modeling the fluid flow in the atmosphere - Solve Navier-Stokes equations - Roughly 100 Flops per grid point with 1 minute timestep • Computational requirements: - To match real-time, need 5 x 1011 flops in 60 seconds = 8 Gflop/s Weather prediction (7 days in 24 hours) 56 Gflop/s Climate prediction (50 years in 30 days) 4. 8 Tflop/s To use in policy negotiations (50 years in 12 hours) 288 Tflop/s • To double the grid resolution / time resolution, computation is 8 x to 16 x • State of the art models require integration of atmosphere, clouds, ocean, sea-ice, land models, plus possibly carbon cycle, geochemistry and more • Current models are coarser than this 24

High Resolution Climate Modeling on NERSC-3 – P. Duffy, et al. , LLNL 25

Which commercial applications require parallelism? Analyzed in detail in “Berkeley View” report www. eecs. berkeley. edu/Pubs/Tech. Rpts/2006/EECS-2006 -183. html • Claim: parallel architecture, language, compiler … must do at least these well to run future parallel apps well • Note: Map. Reduce is embarrassingly parallel; FSM embarrassingly sequential? 26

Outline all • Why powerful computers must be parallel processors Including your laptops and handhelds • Science problems require powerful computers Commercial problems too • Why writing (fast) parallel programs is hard But things are improving • Principles of parallel computing performance • Structure of the course 27

Principles of Parallel Computing • Finding enough parallelism (Amdahl’s Law) • Granularity • Locality • Load balance • Coordination and synchronization • Performance modeling All of these things makes parallel programming even harder than sequential programming. 28

“Automatic” Parallelism in Modern Machines • Bit level parallelism - within floating point operations, etc. • Instruction level parallelism (ILP) - multiple instructions execute per clock cycle • Memory system parallelism - overlap of memory operations with computation • OS parallelism - multiple jobs run in parallel on commodity SMPs Limits to all of these -- for very high performance, need user to identify, schedule and coordinate parallel tasks 29

Amdahl’s Law • Simple software assumption - Fraction F of execution time perfectly parallelizable - No Overhead for – Scheduling – Synchronization – Communication, etc. - Fraction 1 – F Completely Serial • Time on 1 core = (1 – F) / 1 + F / 1 = 1 • Time on N cores = (1 – F) / 1 + F / N

Finding Enough Parallelism 1 Amdahl’s Speedup = 1 -F 1 + F N • Implications: - Attack the common case when introducing parallelization's: When F is small, optimizations will have little effect. - Even if the parallel part speeds up perfectly performance is limited by the sequential part - As N approaches infinity, speedup is bound by 1/(1 – F ). - The aspects you ignore will limit speedup • Discussion: - Can you ever obtain super-linear speedups?

Overhead of Parallelism • Given enough parallel work, this is the biggest barrier to getting desired speedup • Parallelism overheads include: - cost of starting a thread or process cost of communicating shared data cost of synchronizing extra (redundant) computation • Each of these can be in the range of milliseconds (=millions of flops) on some systems • Tradeoff: Algorithm needs sufficiently large units of work to run fast in parallel (i. e. large granularity), but not so large that there is not enough parallel work 32

Locality and Parallelism Conventional Storage Proc Hierarchy Cache L 2 Cache Proc Cache L 2 Cache L 3 Cache Memory potential interconnects L 3 Cache • Large memories are slow, fast memories are small • Storage hierarchies are large and fast on average • Parallel processors, collectively, have large, fast cache - the slow accesses to “remote” data we call “communication” • Algorithm should do most work on local data 33

Processor-DRAM Gap (latency) Goal: find algorithms that minimize communication, not necessarily arithmetic CPU “Moore’s Law” 100 Processor-Memory Performance Gap: (grows 50% / year) DRAM 7%/yr. 10 1 µProc 60%/yr. 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 Performance 1000 Time 34

Load Imbalance • Load imbalance is the time that some processors in the system are idle due to - insufficient parallelism (during that phase) - unequal size tasks • Examples of the latter - adapting to “interesting parts of a domain” - tree-structured computations - fundamentally unstructured problems • Algorithm needs to balance load - Sometimes can determine work load, divide up evenly, before starting - “Static Load Balancing” - Sometimes work load changes dynamically, need to rebalance dynamically - “Dynamic Load Balancing” 35

Building Parallel Software • 2 types of programmers 2 layers • Efficiency Layer (10% of today’s programmers) - Expert programmers build Libraries implementing motifs, “Frameworks”, OS, …. - Highest fraction of peak performance possible • Productivity Layer (90% of today’s programmers) - Domain experts / Naïve programmers productively build parallel applications by composing frameworks & libraries - Hide as many details of machine, parallelism as possible - Willing to sacrifice some performance for productive programming • You may want to work at either level - Goal of this course: understand enough of the efficiency layer to use parallelism effectively 36

Improving Real Performance Peak Performance grows exponentially, a la Moore’s Law 1, 000 But efficiency (the performance relative to the hardware peak) has declined l was 40 -50% on the vector supercomputers of 1990 s Can be as little as 5 -10% on parallel supercomputers of today Close the gap through. . . l l Mathematical methods and algorithms that achieve high performance on a single processor and scale to thousands of processors More efficient programming models and tools for massively parallel supercomputers 100 Teraflops l Peak Performance Gap 10 1 Real Performance 0. 1 1996 2000 2004 37

Outline all • Why powerful computers must be parallel processors Including your laptops and handhelds • Science problems require powerful computers Commercial problems too • Why writing (fast) parallel programs is hard But things are improving • Why hybrid architectures (e. g. , GPU based platforms)? • Structure of the course 38

Hybrid architectures in Top 500 sites

Amdahl’s Law [1967] 1 Amdahl’s Speedup = 1 -F 1 + F N

Designing a milticore platform • Designers must confront single-core design options - Instruction fetch, wakeup, select Execution unit configuration & operand bypass Load/queue(s) & data cache Checkpoint, log, runahead, commit. • As well as additional degrees of freedom - How many cores? How big each? - Shared caches: How many levels? How many banks? - On-chip interconnect: bus, switched?

Want Simple Multicore Hardware Model To Complement Amdahl’s Simple Software Model (1) Chip Hardware Roughly Partitioned into - (I) Multiple Cores (with L 1 caches) - (II) The Rest (L 2/L 3 cache banks, interconnect, pads, etc. ) - Assume: Changing Core Size/Number does NOT change The Rest (2) Resources for Multiple Cores Bounded - Bound of N resources per chip for cores - Due to area, power, cost ($$$), or multiple factors - Bound = Power? (but pictures here use Area)

Simple Multicore Hardware Model, (cont) (3) Architects can improve single-core performance using more of the bounded resource • A Simple Base Core - Consumes 1 Base Core Equivalent (BCE) resources - Provides performance normalized to 1 • An Enhanced Core (in same process generation) - Consumes R x BCEs - Performance as a function or R: Perf(R) • What does function Perf(R) look like?

More on Enhanced Cores • (Performance Perf(R) consuming R BCEs resources) • If Perf(R) > R Always enhance core - Cost-effectively speedups both sequential & parallel • Therefore, equations assume Perf(R) < R • Graphs Assume Perf(R) = square root of R - 2 x performance for 4 BCEs, 3 x for 9 BCEs, etc. - Why? Models diminishing returns with “no coefficients” • Two options symmetric / asymmetric multicore chips

Q 1: How Many (Symmetric) Cores per Chip? • Each Chip Bounded to N BCEs (for all cores) • Each Core consumes R BCEs • Assume Symmetric Multicore = All Cores Identical • Therefore, N/R Cores per Chip — (N/R)*R = N • For an N = 16 BCE Chip: Sixteen 1 -BCE cores Four 4 -BCE cores One 16 -BCE core

Performance of Symmetric Multicore Chips • Serial Fraction 1 -F uses 1 core at rate Perf(R) • Serial time = (1 – F) / Perf(R) • Parallel Fraction uses N/R cores at rate Perf(R) each • Parallel time = F / (Perf(R) * (N/R)) = F*R / Perf(R)*N • Therefore, w. r. t. one base core: 1 Symmetric Speedup = 1 -F Perf(R) + F*R Perf(R)*N • Implications? Enhanced Cores speed Serial & Parallel

Symmetric Multicore Chip, N = 16 BCEs (16 cores) (8 cores) (4 cores) (2 cores) (1 core) F=0. 5 R=16, Cores=1, Speedup=4 F=0. 5, Opt. Speedup S = 4 = 1/(0. 5/4 + 0. 5*16/(4*16)) Need more parallelism to have multicore optimal!

Symmetric Multicore Chip, N = 16 BCEs F=0. 9, R=2, Cores=8, Speedup=6. 7 F=0. 5 R=16, Cores=1, Speedup=4 At F=0. 9, Multicore optimal, but speedup limited Need to obtain even more parallelism!

Symmetric Multicore Chip, N = 16 BCEs F 1, R=1, Cores=16, Speedup 16 F matters: Amdahl’s Law applies to multicore chips Researchers should target parallelism F first

Symmetric Multicore Chip, N = 16 BCEs Recall F=0. 9, R=2, Cores=8, Speedup=6. 7 As Moore’s Law enables N to go from 16 to 256 BCEs, More core enhancements? More cores? Or both?

Symmetric Multicore Chip, N = 256 BCEs F 1 R=1 (vs. 1) Cores=256 (vs. 16) Speedup=204 (vs. 16) MORE CORES! F=0. 99 R=3 (vs. 1) Cores=85 (vs. 16) Speedup=80 (vs. 13. 9) CORE ENHANCEMENTS & MORE CORES! F=0. 9 R=28 (vs. 2) Cores=9 (vs. 8) Speedup=26. 7 (vs. 6. 7) CORE ENHANCEMENTS! As Moore’s Law increases N, often need enhanced core designs Researcher should target single-core performance too

Outline • Symmetric Multicore Chips • Asymmetric Multicore Chips

Asymmetric (Heterogeneous) Multicore Chips • Symmetric Multicore Required All Cores Equal • Why Not Enhance Some (But Not All) Cores? • For Amdahl’s Simple Software Assumptions - One Enhanced Core - Others are Base Cores • How does this affect our hardware model?

How Many Cores per Asymmetric Chip? • Each Chip Bounded to N BCEs (for all cores) • One R-BCE Core leaves N-R BCEs • Use N-R BCEs for N-R Base Cores • Therefore, 1 + N - R Cores per Chip • For an N = 16 BCE Chip: Symmetric: Four 4 -BCE cores Asymmetric: One 4 -BCE core & Twelve 1 -BCE base cores

Performance of Asymmetric Multicore Chips • Serial Fraction 1 -F same, so time = (1 – F) / Perf(R) • Parallel Fraction F - One core at rate Perf(R) - N-R cores at rate 1 - Parallel time = F / (Perf(R) + N - R) • Therefore, w. r. t. one base core: 1 Asymmetric Speedup = 1 -F Perf(R) + N - R

Asymmetric Multicore Chip, N = 256 BCEs (256 cores) (253 cores) (241 cores) (193 cores) (1 core) Number of Cores = 1 (Enhanced) + 256 – R (Base) How do Asymmetric & Symmetric speedups compare?

Outline all • Why powerful computers must be parallel processors Including your laptops and handhelds • Science problems require powerful computers Commercial problems too • Why writing (fast) parallel programs is hard But things are improving • Why hybrid architectures (e. g. , GPU based platforms)? • Structure of the course 57

Course Mechanics Course page: http: //www. ece. ubc. ca/~matei/EECE 571/ Office hours: after each class or by appointment (email me) Email: matei @ ece. ubc. ca Office: KAIS 4033

EECE 571 R: Course Goals • Primary - Gain understanding of fundamental issues that affect design of: - Parallel applications for massively multicore processors - Survey main current research themes • Secondary - By studying a set of outstanding papers, build knowledge of how to do & present research - Learn how to read papers & evaluate ideas What I’ll Assume You Know • You are familiar with - C programming; programming for traditional multicore machines threading, synchronization - Traditional CPU architectures: caching, pipelines, • If there are things that don’t make sense, ask!

This class • Weekly schedule (tentative) • Note: Dates are tentative - But always on Mondays 5 -8 pm, KAIS 4018 Class structure: • Lectures – 1 h - Technology - Research topic • Paper discussion – 1 h • Projects – 1 h

Administravia: Grading • Paper reviews: 25% • Class participation 15% • Discussion leading: 10% • Project: 50%

Administravia: Paper Reviewing (1) • Goals: - Think of what you read - Expand your knowledge beyond the papers that are assigned - Get used to writing paper reviews • Reviews due by midnight the day before the class • Have an eye on the writing style / Be professional in your writing - Clarity - Beware of traps: learn to use them in writing and detect them in reading - Detect (and stay away from) trivial claims.

Administravia: Paper Reviewing (2) Follow the format provided when relevant. • Summarize the main contribution of the paper • Critique the main contribution: - - - Significance: - Rate the significance of the paper on a scale of 5 (breakthrough), 4 (significant contribution), 3 (modest contribution), 2 (incremental contribution), 1 (no contribution or negative contribution). - More importantly: Explain your rating in a sentence or two. Rate how convincing the methodology is. - Do the claims and conclusions follow from the experiments? - Are the assumptions realistic? - Are the experiments well designed? - Are there different experiments that would be more convincing? - Are there other alternatives the authors should have considered? - (And, of course, is the paper free of methodological errors? ) What is the most important limitation of the approach?

Administravia: Paper Reviewing (3) • What are the two strongest and/or most interesting ideas in the paper? • What are the two most striking weaknesses in the paper? • Name two questions that you would like to ask the authors. • Detail an interesting extension to the work not mentioned in the future work section. • Optional comments on the paper that you’d like to see discussed in class.

Administravia: Discussions Leading • Come prepared! - Prepare a 3 -5 minute summary of the paper - Prepare discussion outline - Prepare questions: - “What if”s - Unclear aspects of the solution proposed - … - Similar ideas in different contexts - Initiate short brainstorming sessions • Leaders do NOT need to submit paper reviews • Main goals: - Keep discussion flowing - Keep discussion relevant - Engage everybody (I’ll have an eye on this, too)

Administravia: Projects • It’s just a class, not your Ph. D. • Aim high! - Goal: With one/two extra months of work you project should be ready to be submitted to a decent conference / publication - It is doable! • Combine with your research if relevant to this course • Get informal approval from all instructors if you overlap final projects: - Don’t sell the same piece of work twice - You can get more than twice as many results with less than twice as much work

Administravia: Project timeline (tentative) • 3 rd week – 3 -slide idea presentation : “What is the (research) question you aim to answer? ” • 5 th week: 2 -page project proposal ----------- 1 -week --- spring reading period • 8 th week: 4 -page Midterm project due - Have a clear image of what’s possible/doable - Report preliminary results - Includes related work • Final week [see schedule] In-class project presentation - Presentation - Demo, if appropriate - 8 -page write-up

Next Class (Mon, 17/01) • Note room change: KAIS 4018 • Discussion of - Project ideas - Papers To do: • Subscribe to mailing list • Volunteers for discussion leaders for class next week

What you should get out of the course In depth understanding of: • When is parallel computing useful? • Overview of programming models (software) and tools for massively parallel processors • Some important parallel applications and the algorithms • Performance analysis and tuning • Exposure to various open research questions 69

Questions?

Extra slides 71

Open research questions (Dongarra)

Open research questions (Hwu) • Accelerate computations that have no good parallel structure today. - E. g. , graph analysis • Cata distriubtions cause catastrophical load imbalances in parallel algorithms - free-scal egraphs, MRI spiral scan, astronomy • Computatios with no data reuse - E. g. , matrix vector multiplcation • Algorithm optimizations that are hard to do today - how do I extract locality and regularity of access

Applications CS 267 focus is here • Applications: 1. What are the apps? 2. What are kernels of apps? • Hardware: 3. What are the HW building blocks? 4. How to connect them? • Programming Model / Systems Software: 5. How to describe apps and kernels? 6. How to program the HW? • Evaluation: 7. How to measure success? (Inspired by a view of the Golden Gate Bridge from Berkeley) 74

Par Lab Research Overview Easy to write correct programs that run efficiently on manycore p p A tio a lic y t i v ti c u r d o Pr Laye cy n e i c i Eff ayer L OS. h Arc Personal Image Hearing, Parallel Speech Health Retrieval Music Browser Motifs/Dwarfs Composition & Coordination Language (C&CL) C&CL Compiler/Interpreter Parallel Libraries Efficiency Languages Parallel Frameworks Sketching Static Verification Type Systems Directed Testing Autotuners Dynamic Legacy Communication & Schedulers Checking Code Synch. Primitives Efficiency Language Compilers Debugging OS Libraries & Services with Replay Legacy OS Hypervisor Multicore/GPGPU RAMP Manycore Correctness ns