CSCE 513 Computer Architecture Lecture 12 Vector MachinesGPUs

CSCE 513 Computer Architecture Lecture 12 Vector Machines/GPUs Overview Topics Readings: n n October 24, 2016 Posix Threads finished Very Long Instruction Word (VLIW) Vector Machines GPU overview

Multi-issue – 2– CSCE 513 Fall 2016

ROB- Multi Issue with Speculation – 3– CSCE 513 Fall 2016

. – 4– https: //computing. llnl. gov/tutorials/pthreads/ CSCE 513 Fall 2016

Top 10 challenges in parallel computing By Michael Wrinn (Intel) In priority order: 1. Finding concurrency in a program - how to help programmers “think parallel”? 2. Scheduling tasks at the right granularity onto the processors of a parallel machine. 3. The data locality problem: associating data with tasks and doing it in a way that our target audience will be able to use correctly. 4. Scalability support in hardware: bandwidth and latencies to memory plus interconnects between processing elements. 5. Scalability support in software: libraries, scalable algorithms, and adaptive runtimes to map high level software onto platform details. – 5– CSCE 513 Fall 2016

6. Synchronization constructs (and protocols) that enable programmers write programs free from deadlock and race conditions. 7. Tools, API’s and methodologies to support the debugging process. 8. Error recovery and support for fault tolerance. 9. Support for good software engineering practices: composability, incremental parallelism, and code reuse. – 6– 10. Support for portable performance. What are the right models (or abstractions) so programmers can write code once and expect it to execute well on the important parallel CSCE 513 Fall 2016 platforms?

. – 7– CSCE 513 Fall 2016

1. Matrix multiply 2. Hello_args 2. c (threaded program) – 8– CSCE 513 Fall 2016

Valgrind VALGRIND(1) NAME valgrind - a suite of tools for debugging and profiling programs SYNOPSIS valgrind [[valgrind] [options]] [your-program] [[your-program-options]] DESCRIPTION Valgrind is a flexible program for debugging and profiling Linux executables. It consists of a core, which provides a synthetic CPU in software, and a series of "tools", each of which is a debugging or profiling tool. The architecture is modular, so that new tools can be created easily and without disturbing the existing structure. For more see /usr/share/doc/valgrind/html/index. html – 9– CSCE 513 Fall 2016

Valgrind tools The following tools are available: • cachegrind is a cache simulator. It can be used to annotate every line of your program with the number of instructions executed and cache misses incurred. • callgrind adds call graph tracing to cachegrind. It can be used to get call counts and inclusive cost for each call happening in your program. In addition to cachegrind, callgrind can annotate threads separately, and every instruction of disassembler output of your program with the number of instructions executed and cache misses incurred. – 10 – • helgrind spots potential race conditions in your program. • lackey is a sample tool that can be used as a template for generating your own tools. After the program terminates, it prints out some basic statistics about the program execution. • massif is a heap profiler. It measures how much heap memory your program uses. • memcheck is a fine-grained memory checker. CSCE 513 Fall 2016

Data-Level Parallelism We call these algorithms data parallel algorithms because their parallelism comes from simultaneous operations across large sets of data, rather than from multiple threads of control. W. Daniel Hillis and Guy L. Steele “Data Parallel Algorithms, ” Comm. ACM (1986) “If you were plowing a field, which would you rather use: two strong oxen or 1024 chickens? ” Seymour Cray, Father of the Supercomputer Patterson, David A. ; Hennessy, John L. (2011 -08 -01). Computer Architecture: A Quantitative Approach (The Morgan Kaufmann Series in Computer Architecture and Design) (Kindle Locations 68206826). Elsevier Science (reference). Kindle Edition. – 11 – CSCE 513 Fall 2016

Vector Architectures Basic idea: n n n Read sets of data elements into “vector registers” Operate on those registers Disperse the results back into memory Registers are controlled by compiler n n – 12 – Used to hide memory latency Leverage memory bandwidth Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Vector Architectures VMIPS Example architecture: VMIPS n n Loosely based on Cray-1 Vector registers l Each register holds a 64 -element, 64 bits/element vector l Register file has 16 read ports and 8 write ports n Vector functional units l Fully pipelined l Data and control hazards are detected n Vector load-store unit l Fully pipelined l One word per clock cycle after initial latency n Scalar registers l 32 general-purpose registers l 32 floating-point registers – 13 – Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Vector Architectures VMIPS Instructions ADDVV. D: add two vectors ADDVS. D: add vector to a scalar LV/SV: vector load and vector store from address Example: DAXPY L. D F 0, a LV MULVS. D LV ADDVV SV ; load scalar a V 1, Rx ; load vector X V 2, V 1, F 0 ; vector-scalar multiply V 3, Ry ; load vector Y V 4, V 2, V 3 ; add Ry, V 4 ; store the result Requires 6 instructions vs. almost 600 for MIPS – 14 – Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Vector Architectures Vector Execution Time Execution time depends on three factors: n n n Length of operand vectors Structural hazards Data dependencies VMIPS functional units consume one element per clock cycle n Execution time is approximately the vector length Convey n – 15 – Set of vector instructions that could potentially execute together Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Sequences with read-after-write dependency hazards can be in the same convoy via chaining Vector Architectures Chimes Chaining n Allows a vector operation to start as soon as the individual elements of its vector source operand become available Chime n n n – 16 – Unit of time to execute one convoy m convoys executes in m chimes For vector length of n, requires m x n clock cycles Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Vector Architectures Example LV V 1, Rx ; load vector X MULVS. D V 2, V 1, F 0 ; vector-scalar multiply LV V 3, Ry ; load vector Y ADDVV. D V 4, V 2, V 3 ; add two vectors SV Ry, V 4 ; store the sum Convoys: 1 LV MULVS. D 2 LV ADDVV. D 3 SV 3 chimes, 2 FP ops per result, cycles per FLOP = 1. 5 – 17 – Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016 For 64 element vectors, requires 64 x 3 = 192 clock cycles

Vector Architectures Challenges Start up time n n Latency of vector functional unit Assume the same as Cray-1 l Floating-point add => 6 clock cycles l Floating-point multiply => 7 clock cycles l Floating-point divide => 20 clock cycles l Vector load => 12 clock cycles Improvements: n n n n – 18 – > 1 element per clock cycle Non-64 wide vectors IF statements in vector code Memory system optimizations to support vector processors Multiple dimensional matrices Sparse matrices Programming a vector computer CSCE 513 Fall 2016

Element n of vector register A is “hardwired” to element n of vector register B n – 19 – Vector Architectures Multiple Lanes Allows for multiple hardware lanes Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Vector Architectures Vector Length Register Vector length not known at compile time? Use Vector Length Register (VLR) Use strip mining for vectors over the maximum length: low = 0; VL = (n % MVL); /*find odd-size piece using modulo op % */ for (j = 0; j <= (n/MVL); j=j+1) { /*outer loop*/ for (i = low; i < (low+VL); i=i+1) /*runs for length VL*/ Y[i] = a * X[i] + Y[i] ; /*main operation*/ low = low + VL; /*start of next vector*/ VL = MVL; /*reset the length to maximum vector length*/ } – 20 – Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Vector Architectures Consider: for (i = 0; i < 64; i=i+1) if (X[i] != 0) X[i] = X[i] – Y[i]; Use vector mask register to “disable” elements: – 21 – LV V 1, Rx ; load vector X into V 1 LV V 2, Ry ; load vector Y L. D F 0, #0 ; load FP zero into F 0 SNEVS. D V 1, F 0 ; sets VM(i) to 1 if V 1(i)!=F 0 SUBVV. D V 1, V 2 ; subtract under vector mask SV Rx, V 1 ; store the result in X Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Memory system must be designed to support high bandwidth for vector loads and stores Vector Architectures Memory Banks Spread accesses across multiple banks n n n Control bank addresses independently Load or store non sequential words Support multiple vector processors sharing the same memory Example: n n – 22 – n 32 processors, each generating 4 loads and 2 stores/cycle Processor cycle time is 2. 167 ns, SRAM cycle time is 15 ns How many memory banks needed? CSCE 513 Fall 2016 Copyright © 2012, Elsevier Inc. All rights reserved.

Vector Architectures Stride Consider: for (i = 0; i < 100; i=i+1) for (j = 0; j < 100; j=j+1) { A[i][j] = 0. 0; for (k = 0; k < 100; k=k+1) A[i][j] = A[i][j] + B[i][k] * D[k][j]; } Must vectorize multiplication of rows of B with columns of D Use non-unit stride Bank conflict (stall) occurs when the same bank is hit faster than bank busy time: – 23 – n #banks / LCM(stride, #banks) < bank busy time Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Vector Architectures Consider: for (i = 0; i < n; i=i+1) A[K[i]] = A[K[i]] + C[M[i]]; Use index vector: – 24 – LV Vk, Rk ; load K LVI Va, (Ra+Vk) ; load A[K[]] LV Vm, Rm ; load M LVI Vc, (Rc+Vm) ; load C[M[]] ADDVV. D Va, Vc ; add them SVI (Ra+Vk), Va ; store A[K[]] Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Compilers can provide feedback to programmers Vector Architectures Programming Vec. Architectures Programmers can provide hints to compiler – 25 – Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Media applications operate on data types narrower than the native word size n Example: disconnect carry chains to “partition” adder Limitations, compared to vector instructions: n n n – 26 – Number of data operands encoded into op code No sophisticated addressing modes (strided, scattergather) No mask registers Copyright © 2012, Elsevier Inc. All rights reserved. SIMD Instruction Set Extensions for Multimedia SIMD Extensions CSCE 513 Fall 2016

Implementations: n Intel MMX (1996) l Eight 8 -bit integer ops or four 16 -bit integer ops n Streaming SIMD Extensions (SSE) (1999) l Eight 16 -bit integer ops l Four 32 -bit integer/fp ops or two 64 -bit integer/fp ops n Advanced Vector Extensions (2010) SIMD Instruction Set Extensions for Multimedia SIMD Implementations l Four 64 -bit integer/fp ops n – 27 – Operands must be consecutive and aligned memory locations Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Example DXPY: L. D MOV MOV DADDIU Loop: MUL. 4 D ADD. 4 D S. 4 D DADDIU DSUBU BNEZ – 28 – F 0, a F 1, F 0 F 2, F 0 F 3, F 0 R 4, Rx, #512 L. 4 D F 4, 0[Rx] F 4, F 0 F 8, 0[Ry] F 8, F 4 0[Ry], F 8 Rx, #32 Ry, #32 R 20, R 4, Rx R 20, Loop ; load scalar a ; copy a into F 1 for SIMD MUL ; copy a into F 2 for SIMD MUL ; copy a into F 3 for SIMD MUL ; last address to load ; load X[i], X[i+1], X[i+2], X[i+3] ; a×X[i], a×X[i+1], a×X[i+2], a×X[i+3] ; load Y[i], Y[i+1], Y[i+2], Y[i+3] ; a×X[i]+Y[i], . . . , a×X[i+3]+Y[i+3] ; store into Y[i], Y[i+1], Y[i+2], Y[i+3] ; increment index to X ; increment index to Y ; compute bound ; check if done Copyright © 2012, Elsevier Inc. All rights reserved. SIMD Instruction Set Extensions for Multimedia Example SIMD Code CSCE 513 Fall 2016

Basic idea: n n Plot peak floating-point throughput as a function of arithmetic intensity Ties together floating-point performance and memory performance for a target machine Arithmetic intensity n – 29 – SIMD Instruction Set Extensions for Multimedia Roofline Performance Model Floating-point operations per byte read Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Attainable GFLOPs/sec Min = (Peak Memory BW × Arithmetic Intensity, Peak Floating Point Perf. ) – 30 – Copyright © 2012, Elsevier Inc. All rights reserved. SIMD Instruction Set Extensions for Multimedia Examples CSCE 513 Fall 2016

Given the hardware invested to do graphics well, how can be supplement it to improve performance of a wider range of applications? Graphical Processing Units Basic idea: n Heterogeneous execution model l CPU is the host, GPU is the device n n n – 31 – Develop a C-like programming language for GPU Unify all forms of GPU parallelism as CUDA thread Programming model is “Single Instruction Multiple Thread” Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

A thread is associated with each data element Threads are organized into blocks Graphical Processing Units Threads and Blocks are organized into a grid GPU hardware handles thread management, not applications or OS – 32 – Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Graphical Processing Units NVIDIA GPU Architecture Similarities to vector machines: n Works well with data-level parallel problems n Scatter-gather transfers Mask registers Large register files n n Differences: n n n – 33 – No scalar processor Uses multithreading to hide memory latency Has many functional units, as opposed to a few deeply pipelined units like a vector processor Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

GTX 480 GPU figure 4. 15 Floor plan of the Fermi GTX 480 GPU. This diagram shows 16 multithreaded SIMD Processors. The Thread Block Scheduler is highlighted on the left. The GTX 480 has 6 GDDR 5 ports, each 64 bits wide, supporting up to 6 GB of capacity. The Host Interface is PCI Express 2. 0 x 16. Giga Thread is the name of the scheduler that distributes thread blocks to Multiprocessors, each of which has its own SIMD Thread Scheduler. – 34 – Copyright © 2011, Elsevier Inc. All rights Reserved. CSCE 513 Fall 2016

Graphical Processing Units Example Multiply two vectors of length 8192 n Code that works over all elements is the grid n Thread blocks break this down into manageable sizes l 512 threads per block n n n – 35 – SIMD instruction executes 32 elements at a time Thus grid size = 16 blocks Block is analogous to a strip-mined vector loop with vector length of 32 Block is assigned to a multithreaded SIMD processor by the thread block scheduler Current-generation GPUs (Fermi) have 7 -15 multithreaded SIMD processors Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Graphical Processing Units Terminology Threads of SIMD instructions n Each has its own PC n Thread scheduler uses scoreboard to dispatch No data dependencies between threads! Keeps track of up to 48 threads of SIMD instructions n n l Hides memory latency Thread block scheduler schedules blocks to SIMD processors Within each SIMD processor: n n – 36 – 32 SIMD lanes Wide and shallow compared to vector processors Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Graphical Processing Units Example NVIDIA GPU has 32, 768 registers n Divided into lanes n Each SIMD thread is limited to 64 registers SIMD thread has up to: n l 64 vector registers of 32 32 -bit elements l 32 vector registers of 32 64 -bit elements n – 37 – Fermi has 16 physical SIMD lanes, each containing 2048 registers Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

ISA is an abstraction of the hardware instruction set n “Parallel Thread Execution (PTX)” n Uses virtual registers Translation to machine code is performed in software Example: n n Graphical Processing Units NVIDIA Instruction Set Arch. shl. s 32 R 8, block. Idx, 9 ; Thread Block ID * Block size (512 or 29) add. s 32 R 8, thread. Idx ; R 8 = i = my CUDA thread ID ld. global. f 64 RD 0, [X+R 8] ; RD 0 = X[i] ld. global. f 64 RD 2, [Y+R 8] ; RD 2 = Y[i] mul. f 64 R 0 D, RD 0, RD 4 ; Product in RD 0 = RD 0 * RD 4 (scalar a) add. f 64 R 0 D, RD 0, RD 2 ; Sum in RD 0 = RD 0 + RD 2 (Y[i]) st. global. f 64 [Y+R 8], RD 0 ; Y[i] = sum (X[i]*a + Y[i]) – 38 – Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Like vector architectures, GPU branch hardware uses internal masks Graphical Processing Units Conditional Branching Also uses n Branch synchronization stack l Entries consist of masks for each SIMD lane l I. e. which threads commit their results (all threads execute) n Instruction markers to manage when a branch diverges into multiple execution paths l Push on divergent branch n …and when paths converge l Act as barriers l Pops stack Per-thread-lane 1 -bit predicate register, specified by programmer – 39 – Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Graphical Processing Units Example if (X[i] != 0) X[i] = X[i] – Y[i]; else X[i] = Z[i]; ld. global. f 64 RD 0, [X+R 8] ; RD 0 = X[i] setp. neq. s 32 P 1, RD 0, #0 ; P 1 is predicate register 1 @!P 1, bra bits ELSE 1, *Push ; Push old mask, set new mask ; if P 1 false, go to ELSE 1 ld. global. f 64 RD 2, [Y+R 8] ; RD 2 = Y[i] sub. f 64 RD 0, RD 2 ; Difference in RD 0 st. global. f 64 [X+R 8], RD 0 ; X[i] = RD 0 @P 1, bra ENDIF 1, *Comp ; complement mask bits ; if P 1 true, go to ENDIF 1 ELSE 1: – 40 – ld. global. f 64 RD 0, [Z+R 8] st. global. f 64 [X+R 8], Copyright © 2012, Elsevier Inc. All rights reserved. RD 0 ; RD 0 = Z[i] ; X[i] = RD 0 CSCE 513 Fall 2016

Each SIMD Lane has private section of off-chip DRAM n n Graphical Processing Units NVIDIA GPU Memory Structures “Private memory” Contains stack frame, spilling registers, and private variables Each multithreaded SIMD processor also has local memory n Shared by SIMD lanes / threads within a block Memory shared by SIMD processors is GPU Memory n – 41 – Host can read and write GPU memory Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Each SIMD processor has n n n Two SIMD thread schedulers, two instruction dispatch units 16 SIMD lanes (SIMD width=32, chime=2 cycles), 16 loadstore units, 4 special function units Thus, two threads of SIMD instructions are scheduled every two clock cycles Graphical Processing Units Fermi Architecture Innovations Fast double precision Caches for GPU memory 64 -bit addressing and unified address space Error correcting codes Faster context switching – 42 – Faster atomic instructions Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

– 43 – Copyright © 2012, Elsevier Inc. All rights reserved. Graphical Processing Units Fermi Multithreaded SIMD Proc. CSCE 513 Fall 2016

Focuses on determining whether data accesses in later iterations are dependent on data values produced in earlier iterations n Loop-carried dependence Example 1: Detecting and Enhancing Loop-Level Parallelism for (i=999; i>=0; i=i-1) x[i] = x[i] + s; No loop-carried dependence – 44 – Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Detecting and Enhancing Loop-Level Parallelism Example 2: for (i=0; i<100; i=i+1) { A[i+1] = A[i] + C[i]; /* S 1 */ B[i+1] = B[i] + A[i+1]; /* S 2 */ } S 1 and S 2 use values computed by S 1 in previous iteration S 2 uses value computed by S 1 in same iteration – 45 – Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Example 3: for (i=0; i<100; i=i+1) { A[i] = A[i] + B[i]; /* S 1 */ B[i+1] = C[i] + D[i]; /* S 2 */ } S 1 uses value computed by S 2 in previous iteration but dependence is not circular so loop is parallel Transform to: A[0] = A[0] + B[0]; for (i=0; i<99; i=i+1) { B[i+1] = C[i] + D[i]; A[i+1] = A[i+1] + B[i+1]; } B[100] = C[99] + D[99]; – 46 – Copyright © 2012, Elsevier Inc. All rights reserved. Detecting and Enhancing Loop-Level Parallelism CSCE 513 Fall 2016

Example 4: for (i=0; i<100; i=i+1) { A[i] = B[i] + C[i]; D[i] = A[i] * E[i]; } Detecting and Enhancing Loop-Level Parallelism Example 5: for (i=1; i<100; i=i+1) { Y[i] = Y[i-1] + Y[i]; } – 47 – Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Detecting and Enhancing Loop-Level Parallelism Finding dependencies Assume indices are affine: n a x i + b (i is loop index) Assume: n Store to a x i + b, then Load from c x i + d i runs from m to n n Dependence exists if: n n l Given j, k such that m ≤ j ≤ n, m ≤ k ≤ n l Store to a x j + b, load from a x k + d, and a x j + b = c x k + d – 48 – Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Generally cannot determine at compile time Test for absence of a dependence: n GCD test: l If a dependency exists, GCD(c, a) must evenly divide (d-b) Example: Detecting and Enhancing Loop-Level Parallelism Finding dependencies for (i=0; i<100; i=i+1) { X[2*i+3] = X[2*i] * 5. 0; } – 49 – Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Example 2: for (i=0; i<100; i=i+1) { Y[i] = X[i] / c; /* S 1 */ X[i] = X[i] + c; /* S 2 */ Z[i] = Y[i] + c; /* S 3 */ Y[i] = c - Y[i]; /* S 4 */ Detecting and Enhancing Loop-Level Parallelism Finding dependencies } Watch for antidependencies and output dependencies – 50 – Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Example 2: for (i=0; i<100; i=i+1) { Y[i] = X[i] / c; /* S 1 */ X[i] = X[i] + c; /* S 2 */ Z[i] = Y[i] + c; /* S 3 */ Y[i] = c - Y[i]; /* S 4 */ Detecting and Enhancing Loop-Level Parallelism Finding dependencies } Watch for antidependencies and output dependencies – 51 – Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016

Reduction Operation: for (i=9999; i>=0; i=i-1) sum = sum + x[i] * y[i]; Transforms to… for (i=9999; i>=0; i=i-1) sum [i] = x[i] * y[i]; for (i=9999; i>=0; i=i-1) Detecting and Enhancing Loop-Level Parallelism Reductions finalsum = finalsum + sum[i]; Do on p processors: for (i=999; i>=0; i=i-1) finalsum[p] = finalsum[p] + sum[i+1000*p]; Note: assumes associativity! – 52 – Copyright © 2012, Elsevier Inc. All rights reserved. CSCE 513 Fall 2016