Computer Architecture A Quantitative Approach Sixth Edition Chapter

Computer Architecture A Quantitative Approach, Sixth Edition Chapter 4 Data-Level Parallelism in Vector, SIMD, and GPU Architectures Copyright © 2019, Elsevier Inc. All rights Reserved 1

n SIMD architectures can exploit significant datalevel parallelism for: n n n Matrix-oriented scientific computing Media-oriented image and sound processors SIMD is more energy efficient than MIMD n n n Introduction Only needs to fetch one instruction per data operation Makes SIMD attractive for personal mobile devices SIMD allows programmer to continue to think sequentially Copyright © 2019, Elsevier Inc. All rights Reserved 2

n Vector architectures SIMD extensions Graphics Processor Units (GPUs) n For x 86 processors: n n n Introduction SIMD Parallelism Expect two additional cores per chip per year SIMD width to double every four years Potential speedup from SIMD to be twice that from MIMD! Copyright © 2019, Elsevier Inc. All rights Reserved 3

n Basic idea: n n Read sets of data elements into “vector registers” Operate on those registers Disperse the results back into memory Vector Architectures Registers are controlled by compiler n n Used to hide memory latency Leverage memory bandwidth Copyright © 2019, Elsevier Inc. All rights Reserved 4

n Example architecture: RV 64 V n n Loosely based on Cray-1 32 62 -bit vector registers n n n Fully pipelined Data and control hazards are detected Vector load-store unit n n n Register file has 16 read ports and 8 write ports Vector functional units n n Vector Architectures VMIPS Fully pipelined One word per clock cycle after initial latency Scalar registers n n 31 general-purpose registers 32 floating-point registers Copyright © 2019, Elsevier Inc. All rights Reserved 5

n n n . vv: two vector operands. vs and. sv: vector and scalar operands LV/SV: vector load and vector store from address Example: DAXPY vsetdcfg 4*FP 64 # Enable 4 DP FP vregs fld f 0, a # Load scalar a vld v 0, x 5 # Load vector X vmul v 1, v 0, f 0 # Vector-scalar mult vld v 2, x 6 # Load vector Y vadd v 3, v 1, v 2 # Vector-vector add vst v 3, x 6 # Store the sum vdisable # Disable vector regs 8 instructions, 258 for RV 64 V (scalar code) Copyright © 2019, Elsevier Inc. All rights Reserved Vector Architectures VMIPS Instructions 6

n Execution time depends on three factors: n n RV 64 V functional units consume one element per clock cycle n n Length of operand vectors Structural hazards Data dependencies Vector Architectures Vector Execution Time Execution time is approximately the vector length Convey n Set of vector instructions that could potentially execute together Copyright © 2019, Elsevier Inc. All rights Reserved 7

n n Sequences with read-after-write dependency hazards placed in same convey via chaining Chaining n n Vector Architectures Chimes Allows a vector operation to start as soon as the individual elements of its vector source operand become available Chime n n n Unit of time to execute one convey m conveys executes in m chimes for vector length n For vector length of n, requires m x n clock cycles Copyright © 2019, Elsevier Inc. All rights Reserved 8

vld vmul vld vadd vst v 0, x 5 v 1, v 0, f 0 v 2, x 6 v 3, v 1, v 2 v 3, x 6 Convoys: 1 vld 2 vld 3 vst # Load vector X # Vector-scalar multiply # Load vector Y # Vector-vector add # Store the sum Vector Architectures Example vmul vadd 3 chimes, 2 FP ops per result, cycles per FLOP = 1. 5 For 64 element vectors, requires 32 x 3 = 96 clock cycles Copyright © 2019, Elsevier Inc. All rights Reserved 9

n Start up time n n Latency of vector functional unit Assume the same as Cray-1 n n n Vector Architectures Challenges Floating-point add => 6 clock cycles Floating-point multiply => 7 clock cycles Floating-point divide => 20 clock cycles Vector load => 12 clock cycles Improvements: n n n n > 1 element per clock cycle Non-64 wide vectors IF statements in vector code Memory system optimizations to support vector processors Multiple dimensional matrices Sparse matrices Programming a vector computer Copyright © 2019, Elsevier Inc. All rights Reserved 10

n Element n of vector register A is “hardwired” to element n of vector register B n Allows for multiple hardware lanes Copyright © 2019, Elsevier Inc. All rights Reserved Vector Architectures Multiple Lanes 11

for (i=0; i <n; i=i+1) Y[i] = a * X[i] + Y[i]; vsetdcfg 2 DP FP # Enable 2 64 b Fl. Pt. registers fld f 0, a # Load scalar a loop: setvl t 0, a 0 # vl = t 0 = min(mvl, n) vld v 0, x 5 # Load vector X slli t 1, t 0, 3 # t 1 = vl * 8 (in bytes) add x 5, t 1 # Increment pointer to X by vl*8 vmul v 0, f 0 # Vector-scalar mult vld v 1, x 6 # Load vector Y vadd v 1, v 0, v 1 # Vector-vector add sub a 0, t 0 # n -= vl (t 0) vst v 1, x 6 # Store the sum into Y add x 6, t 1 # Increment pointer to Y by vl*8 bnez a 0, loop # Repeat if n != 0 vdisable # Disable vector regs} Copyright © 2019, Elsevier Inc. All rights Reserved Vector Architectures Vector Length Register 12

n n Consider: for (i = 0; i < 64; i=i+1) if (X[i] != 0) X[i] = X[i] – Y[i]; Use predicate register to “disable” elements: vsetdcfg vsetpcfgi vld fmv. d. x vpne vsub vst vdisable vpdisable 2*FP 64 1 v 0, x 5 v 1, x 6 f 0, x 0 p 0, v 0, f 0 v 0, v 1 v 0, x 5 Vector Architectures Vector Mask Registers # Enable 2 64 b FP vector regs # Enable 1 predicate register # Load vector X into v 0 # Load vector Y into v 1 # Put (FP) zero into f 0 # Set p 0(i) to 1 if v 0(i)!=f 0 # Subtract under vector mask # Store the result in X # Disable vector registers # Disable predicate registers Copyright © 2019, Elsevier Inc. All rights Reserved 13

n n Memory system must be designed to support high bandwidth for vector loads and stores Spread accesses across multiple banks n n Vector Architectures Memory Banks Control bank addresses independently Load or store non sequential words (need independent bank addressing) Support multiple vector processors sharing the same memory Example: n n n 32 processors, each generating 4 loads and 2 stores/cycle Processor cycle time is 2. 167 ns, SRAM cycle time is 15 ns How many memory banks needed? n 32 x(4+2)x 15/2. 167 = ~1330 banks Copyright © 2019, Elsevier Inc. All rights Reserved 14

n n Consider: for (i = 0; i < 100; i=i+1) for (j = 0; j < 100; j=j+1) { A[i][j] = 0. 0; for (k = 0; k < 100; k=k+1) A[i][j] = A[i][j] + B[i][k] * D[k][j]; } Vector Architectures Stride Must vectorize multiplication of rows of B with columns of D Use non-unit stride Bank conflict (stall) occurs when the same bank is hit faster than bank busy time: n #banks / LCM(stride, #banks) < bank busy time Copyright © 2019, Elsevier Inc. All rights Reserved 15

Consider: for (i = 0; i < n; i=i+1) A[K[i]] = A[K[i]] + C[M[i]]; n Use index vector: vsetdcfg 4*FP 64 # 4 64 b FP vector registers vld v 0, x 7 # Load K[] vldx v 1, x 5, v 0 # Load A[K[]] vld v 2, x 28 # Load M[] vldi v 3, x 6, v 2 # Load C[M[]] vadd v 1, v 3 # Add them vstx v 1, x 5, v 0 # Store A[K[]] vdisable # Disable vector registers n Copyright © 2019, Elsevier Inc. All rights Reserved Vector Architectures Scatter-Gather 16

n n Compilers can provide feedback to programmers Programmers can provide hints to compiler Copyright © 2019, Elsevier Inc. All rights Reserved Vector Architectures Programming Vec. Architectures 17

n n Media applications operate on data types narrower than the native word size n Example: disconnect carry chains to “partition” adder Limitations, compared to vector instructions: n Number of data operands encoded into op code n No sophisticated addressing modes (strided, scattergather) n No mask registers Copyright © 2019, Elsevier Inc. All rights Reserved SIMD Instruction Set Extensions for Multimedia SIMD Extensions 18

n Implementations: n Intel MMX (1996) n n Streaming SIMD Extensions (SSE) (1999) n n n Four 64 -bit integer/fp ops AVX-512 (2017) n n Eight 16 -bit integer ops Four 32 -bit integer/fp ops or two 64 -bit integer/fp ops Advanced Vector Extensions (2010) n n Eight 8 -bit integer ops or four 16 -bit integer ops SIMD Instruction Set Extensions for Multimedia SIMD Implementations Eight 64 -bit integer/fp ops Operands must be consecutive and aligned memory locations Copyright © 2019, Elsevier Inc. All rights Reserved 19

n Example DXPY: fld splat. 4 D addi Loop: fld. 4 D fmul. 4 D fld. 4 D fadd. 4 D fsd. 4 D addi bne f 0, a f 0, f 0 x 28, x 5, #256 f 1, 0(x 5) f 1, f 0 f 2, 0(x 6) f 2, f 1 f 2, 0(x 6) x 5, #32 x 6, #32 x 28, x 5, Loop # Load scalar a # Make 4 copies of a # Last address to load # Load X[i]. . . X[i+3] # a x X[i]. . . a x X[i+3] # Load Y[i]. . . Y[i+3] # a x X[i]+Y[i]. . . # a x X[i+3]+Y[i+3] # Store Y[i]. . . Y[i+3] # Increment index to X # Increment index to Y # Check if done Copyright © 2019, Elsevier Inc. All rights Reserved SIMD Instruction Set Extensions for Multimedia Example SIMD Code 20

n n Basic idea: n Plot peak floating-point throughput as a function of arithmetic intensity n Ties together floating-point performance and memory performance for a target machine Arithmetic intensity n Floating-point operations per byte read Copyright © 2019, Elsevier Inc. All rights Reserved SIMD Instruction Set Extensions for Multimedia Roofline Performance Model 21

n Attainable GFLOPs/sec = (Peak Memory BW × Arithmetic Intensity, Peak Floating Point Perf. ) Copyright © 2019, Elsevier Inc. All rights Reserved SIMD Instruction Set Extensions for Multimedia Examples 22

n Basic idea: n Heterogeneous execution model n n CPU is the host, GPU is the device Develop a C-like programming language for GPU Unify all forms of GPU parallelism as CUDA thread Programming model is “Single Instruction Multiple Thread” Copyright © 2019, Elsevier Inc. All rights Reserved Graphical Processing Units 23

Graphical Processing Units Threads and Blocks n n A thread is associated with each data element Threads are organized into blocks Blocks are organized into a grid GPU hardware handles thread management, not applications or OS Copyright © 2019, Elsevier Inc. All rights Reserved 24

n Similarities to vector machines: n n n Works well with data-level parallel problems Scatter-gather transfers Mask registers Large register files Graphical Processing Units NVIDIA GPU Architecture Differences: n n n No scalar processor Uses multithreading to hide memory latency Has many functional units, as opposed to a few deeply pipelined units like a vector processor Copyright © 2019, Elsevier Inc. All rights Reserved 25

n n Code that works over all elements is the grid Thread blocks break this down into manageable sizes n n n 512 threads per block SIMD instruction executes 32 elements at a time Thus grid size = 16 blocks Block is analogous to a strip-mined vector loop with vector length of 32 Block is assigned to a multithreaded SIMD processor by the thread block scheduler Current-generation GPUs have 7 -15 multithreaded SIMD processors Copyright © 2019, Elsevier Inc. All rights Reserved Graphical Processing Units Example 26

n n Each thread is limited to 64 registers Groups of 32 threads combined into a SIMD thread or “warp” n n Up to 32 warps are scheduled on a single SIMD processor n n n Mapped to 16 physical lanes Each warp has its own PC Thread scheduler uses scoreboard to dispatch warps By definition, no data dependencies between warps Dispatch warps into pipeline, hide memory latency Graphical Processing Units Terminology Thread block scheduler schedules blocks to SIMD processors Within each SIMD processor: n n 32 SIMD lanes Wide and shallow compared to vector processors Copyright © 2019, Elsevier Inc. All rights Reserved 27

Graphical Processing Units Example Copyright © 2019, Elsevier Inc. All rights Reserved 28

Copyright © 2019, Elsevier Inc. All rights Reserved Graphical Processing Units GPU Organization 29

n ISA is an abstraction of the hardware instruction set n “Parallel Thread Execution (PTX)” n n opcode. type d, a, b, c; Uses virtual registers Translation to machine code is performed in software Example: Graphical Processing Units NVIDIA Instruction Set Arch. shl. s 32 R 8, block. Idx, 9 ; Thread Block ID * Block size (512 or 29) add. s 32 R 8, thread. Idx ; R 8 = i = my CUDA thread ID ld. global. f 64 RD 0, [X+R 8] ; RD 0 = X[i] ld. global. f 64 RD 2, [Y+R 8] ; RD 2 = Y[i] mul. f 64 R 0 D, RD 0, RD 4 ; Product in RD 0 = RD 0 * RD 4 (scalar a) add. f 64 R 0 D, RD 0, RD 2 ; Sum in RD 0 = RD 0 + RD 2 (Y[i]) st. global. f 64 [Y+R 8], RD 0 ; Y[i] = sum (X[i]*a + Y[i]) Copyright © 2019, Elsevier Inc. All rights Reserved 30

n n Like vector architectures, GPU branch hardware uses internal masks Also uses n Branch synchronization stack n n n Instruction markers to manage when a branch diverges into multiple execution paths n n Push on divergent branch …and when paths converge n n n Entries consist of masks for each SIMD lane I. e. which threads commit their results (all threads execute) Graphical Processing Units Conditional Branching Act as barriers Pops stack Per-thread-lane 1 -bit predicate register, specified by programmer Copyright © 2019, Elsevier Inc. All rights Reserved 31

if (X[i] != 0) X[i] = X[i] – Y[i]; else X[i] = Z[i]; ld. global. f 64 setp. neq. s 32 @!P 1, bra RD 0, [X+R 8] P 1, RD 0, #0 ELSE 1, *Push ; RD 0 = X[i] ; P 1 is predicate register 1 ; Push old mask, set new mask bits ; if P 1 false, go to ELSE 1 ld. global. f 64 RD 2, [Y+R 8] ; RD 2 = Y[i] sub. f 64 RD 0, RD 2 ; Difference in RD 0 st. global. f 64 [X+R 8], RD 0 ; X[i] = RD 0 @P 1, bra ENDIF 1, *Comp ; complement mask bits ; if P 1 true, go to ENDIF 1 ELSE 1: ld. global. f 64 RD 0, [Z+R 8] ; RD 0 = Z[i] st. global. f 64 [X+R 8], RD 0 ; X[i] = RD 0 ENDIF 1: <next instruction>, *Pop ; pop to restore old mask Copyright © 2019, Elsevier Inc. All rights Reserved Graphical Processing Units Example 32

n n Each SIMD Lane has private section of off-chip DRAM n “Private memory” n Contains stack frame, spilling registers, and private variables Each multithreaded SIMD processor also has local memory n n Graphical Processing Units NVIDIA GPU Memory Structures Shared by SIMD lanes / threads within a block Memory shared by SIMD processors is GPU Memory n Host can read and write GPU memory Copyright © 2019, Elsevier Inc. All rights Reserved 33

n Each SIMD processor has n n n n Two or four SIMD thread schedulers, two instruction dispatch units 16 SIMD lanes (SIMD width=32, chime=2 cycles), 16 load-store units, 4 special function units Two threads of SIMD instructions are scheduled every two clock cycles Graphical Processing Units Pascal Architecture Innovations Fast single-, double-, and half-precision High Bandwith Memory 2 (HBM 2) at 732 GB/s NVLink between multiple GPUs (20 GB/s in each direction) Unified virtual memory and paging support Copyright © 2019, Elsevier Inc. All rights Reserved 34

Copyright © 2019, Elsevier Inc. All rights Reserved Graphical Processing Units Pascal Multithreaded SIMD Proc. 35

n n SIMD processor analogous to vector processor, both have MIMD Registers n n n n RV 64 V register file holds entire vectors GPU distributes vectors across the registers of SIMD lanes RV 64 has 32 vector registers of 32 elements (1024) GPU has 256 registers with 32 elements each (8 K) RV 64 has 2 to 8 lanes with vector length of 32, chime is 4 to 16 cycles SIMD processor chime is 2 to 4 cycles GPU vectorized loop is grid All GPU loads are gather instructions and all GPU stores are scatter instructions Copyright © 2019, Elsevier Inc. All rights Reserved Graphical Processing Units Vector Architectures vs GPUs 36

n n n GPUs have more SIMD lanes GPUs have hardware support for more threads Both have 2: 1 ratio between double- and single-precision performance Both have 64 -bit addresses, but GPUs have smaller memory SIMD architectures have no scatter-gather support Copyright © 2019, Elsevier Inc. All rights Reserved Graphical Processing Units SIMD Architectures vs GPUs 37

n Focuses on determining whether data accesses in later iterations are dependent on data values produced in earlier iterations n n Loop-carried dependence Example 1: for (i=999; i>=0; i=i-1) x[i] = x[i] + s; n Detecting and Enhancing Loop-Level Parallelism No loop-carried dependence Copyright © 2019, Elsevier Inc. All rights Reserved 38

n Example 2: for (i=0; i<100; i=i+1) { A[i+1] = A[i] + C[i]; /* S 1 */ B[i+1] = B[i] + A[i+1]; /* S 2 */ } n n S 1 and S 2 use values computed by S 1 in previous iteration S 2 uses value computed by S 1 in same iteration Copyright © 2019, Elsevier Inc. All rights Reserved Detecting and Enhancing Loop-Level Parallelism 39

n n n Example 3: for (i=0; i<100; i=i+1) { A[i] = A[i] + B[i]; /* S 1 */ B[i+1] = C[i] + D[i]; /* S 2 */ } S 1 uses value computed by S 2 in previous iteration but dependence is not circular so loop is parallel Transform to: A[0] = A[0] + B[0]; for (i=0; i<99; i=i+1) { B[i+1] = C[i] + D[i]; A[i+1] = A[i+1] + B[i+1]; } B[100] = C[99] + D[99]; Copyright © 2019, Elsevier Inc. All rights Reserved Detecting and Enhancing Loop-Level Parallelism 40

n n Example 4: for (i=0; i<100; i=i+1) { A[i] = B[i] + C[i]; D[i] = A[i] * E[i]; } Example 5: for (i=1; i<100; i=i+1) { Y[i] = Y[i-1] + Y[i]; } Copyright © 2019, Elsevier Inc. All rights Reserved Detecting and Enhancing Loop-Level Parallelism 41

n Assume indices are affine: n n a x i + b (i is loop index) Assume: n n Store to a x i + b, then Load from c x i + d i runs from m to n Dependence exists if: n n Given j, k such that m ≤ j ≤ n, m ≤ k ≤ n Store to a x j + b, load from a x k + d, and a x j + b = c x k + d Copyright © 2019, Elsevier Inc. All rights Reserved Detecting and Enhancing Loop-Level Parallelism Finding dependencies 42

n n Generally cannot determine at compile time Test for absence of a dependence: n GCD test: n n If a dependency exists, GCD(c, a) must evenly divide (d-b) Example: for (i=0; i<100; i=i+1) { X[2*i+3] = X[2*i] * 5. 0; } Copyright © 2019, Elsevier Inc. All rights Reserved Detecting and Enhancing Loop-Level Parallelism Finding dependencies 43

n Example 2: for (i=0; i<100; i=i+1) { Y[i] = X[i] / c; /* S 1 */ X[i] = X[i] + c; /* S 2 */ Z[i] = Y[i] + c; /* S 3 */ Y[i] = c - Y[i]; /* S 4 */ } n Watch for antidependencies and output dependencies Copyright © 2019, Elsevier Inc. All rights Reserved Detecting and Enhancing Loop-Level Parallelism Finding dependencies 44

n Example 2: for (i=0; i<100; i=i+1) { Y[i] = X[i] / c; /* S 1 */ X[i] = X[i] + c; /* S 2 */ Z[i] = Y[i] + c; /* S 3 */ Y[i] = c - Y[i]; /* S 4 */ } n Watch for antidependencies and output dependencies Copyright © 2019, Elsevier Inc. All rights Reserved Detecting and Enhancing Loop-Level Parallelism Finding dependencies 45

n n Reduction Operation: for (i=9999; i>=0; i=i-1) sum = sum + x[i] * y[i]; Transform to… for (i=9999; i>=0; i=i-1) sum [i] = x[i] * y[i]; for (i=9999; i>=0; i=i-1) finalsum = finalsum + sum[i]; Do on p processors: for (i=999; i>=0; i=i-1) finalsum[p] = finalsum[p] + sum[i+1000*p]; Note: assumes associativity! Copyright © 2019, Elsevier Inc. All rights Reserved Detecting and Enhancing Loop-Level Parallelism Reductions 46

n GPUs suffer from being coprocessors n n Concentrating on peak performance in vector architectures and ignoring start-up overhead n n GPUs have flexibility to change ISA Fallacies and Pitfalls Overheads require long vector lengths to achieve speedup Increasing vector performance without comparable increases in scalar performance You can get good vector performance without providing memory bandwidth On GPUs, just add more threads if you don’t have enough memory performance Copyright © 2019, Elsevier Inc. All rights Reserved 47