Exploiting Parallelism We can exploit parallelism in two

Exploiting Parallelism at the Instruction level (SIMD) § Consider adding together two arrays: void

Intel SSE/SSE 2 as an example of SIMD • Added new 128 bit registers

SIMD Extensions More than 70 instructions. Arithmetic Operations supported: Addition, Subtraction, Mult, Division, Square

Is it always that easy? § No, not always. Let’s look at a little

$We first need to restructure the code unsigned sum_array 2(unsigned *array, int length) {$

Then we can write SIMD code for the hot part unsigned sum_array 2(unsigned *array,

Exploiting Parallelism at the core level § Consider the code from the second practice

Open. MP § Many APIs exist/being developed to exploit multiple cores — Open. MP

Race conditions § If we parallelize the inner loop, the code produces wrong answers:

Another example § Assume a[N][N] and b[N] are integer arrays as before: int total

Let’s try the version with race conditions § We’re expecting a significant speedup on

This mental picture is wrong! § We’ve forgotten about caches! — The memory may

Summary § Performance is of primary concern in some applications — Games, servers, mobile

Cachegrind comparison § I refs: I 1 misses: L 2 i misses: I 1

Slides: 17

Download presentation

Exploiting Parallelism § We can exploit parallelism in two ways: 1. At the instruction level — Single Instruction Multiple Data (SIMD) — Make use of extensions to the ISA 2. At the core level — Rewrite the code to parallelize operations across many cores — Make use of extensions to the programming language 1

Exploiting Parallelism at the Instruction level (SIMD) § Consider adding together two arrays: void array_add(int A[], int B[], int C[], int length) { int i; for (i = 0; i < length; ++i) { C[i] = A[i] + B[i]; } } Operating on one element at a time + 2

Intel SSE/SSE 2 as an example of SIMD • Added new 128 bit registers (XMM 0 – XMM 7), each can store — — — + 4 single precision FP values (SSE) 2 double precision FP values (SSE 2) 16 byte values (SSE 2) 8 word values (SSE 2) 4 double word values (SSE 2) 1 128 -bit integer value (SSE 2) 4 * 32 b 2 * 64 b 16 * 8 b 8 * 16 b 4 * 32 b 1 * 128 b 4. 0 (32 bits) 3. 5 (32 bits) -2. 0 (32 bits) -1. 5 (32 bits) 2. 0 (32 bits) 1. 7 (32 bits) 2. 3 (32 bits) 2. 5 (32 bits) 6. 0 (32 bits) 5. 2 (32 bits) 0. 3 (32 bits) 5

SIMD Extensions More than 70 instructions. Arithmetic Operations supported: Addition, Subtraction, Mult, Division, Square Root, Maximum, Minimum. Can operate on Floating point or Integer data. 6

Is it always that easy? § No, not always. Let’s look at a little more challenging one: unsigned sum_array(unsigned *array, int length) { int total = 0; for (int i = 0; i < length; ++i) { total += array[i]; } return total; } § Can we exploit SIMD-style parallelism? 7

$We first need to restructure the code unsigned sum_array 2(unsigned *array, int length) {$

We first need to restructure the code unsigned sum_array 2(unsigned *array, int length) { unsigned total, i; unsigned temp[4] = {0, 0, 0, 0}; for (i = 0; i < length & ~0 x 3; i += 4) { temp[0] += array[i]; temp[1] += array[i+1]; temp[2] += array[i+2]; temp[3] += array[i+3]; } total = temp[0] + temp[1] + temp[2] + temp[3]; for ( ; i < length; ++i) { total += array[i]; } return total; }

Then we can write SIMD code for the hot part unsigned sum_array 2(unsigned *array, int length) { unsigned total, i; unsigned temp[4] = {0, 0, 0, 0}; for (i = 0; i < length & ~0 x 3; i += 4) { temp[0] += array[i]; temp[1] += array[i+1]; temp[2] += array[i+2]; temp[3] += array[i+3]; } total = temp[0] + temp[1] + temp[2] + temp[3]; for ( ; i < length; ++i) { total += array[i]; } return total; } 9

Exploiting Parallelism at the core level § Consider the code from the second practice Midterm: for(int i = 0; i < N; ++i) { c[i] = 0; for(int j = 0; j < N; ++j) c[i] += a[i][j] * b[j]; } parallel_for split the iterations across multiple cores (fork/join) 1. If there is a choice, what should we parallelize? Inner loop? Outer? Both? row i c[i] 2. How are N iterations split across c cores? Core 0 Core 1 [0, N/c) [N/c, 2 N/c) Core 0 Core 1 0, c, 2 c, … 1, c+1, 2 c+1, … . . . 10

Open. MP § Many APIs exist/being developed to exploit multiple cores — Open. MP is easy to use, but has limitations #include <omp. h> #pragma omp parallel for(int i = 0; i < N; ++i) { c[i] = 0; for(int j = 0; j < N; ++j) c[i] += a[i][j] * b[j]; } § Compile your code with: g++ –fopenmp mp 6. cxx § Google for Open. MP LLNL for a handy reference-page 11

Race conditions § If we parallelize the inner loop, the code produces wrong answers: for(int i = 0; i < N; ++i) { c[i] = 0; #pragma omp parallel for(int j = 0; j < N; ++j) c[i] += a[i][j] * b[j]; } § § 1. load c[i] In each iteration, each thread does: 2. compute new c[i] 3. update c[i] thread 1 thread 2 What if this happens: 1. load c[i] 2. load c[i] 3. compute c[i] 4. compute c[i] data race 5. update c[i] 6. update c[i] 12

Another example § Assume a[N][N] and b[N] are integer arrays as before: int total = 0; #pragma omp parallel for (plus a bit more!) for(int i = 0; i < N; ++i) { for(int j = 0; j < N; ++j) total += a[i][j] * b[j]; } § Which loop should we parallelize? § Answer: Outer loop, if we can avoid the race condition — Each thread maintains a private copy of total (initialized to zero) — When the threads are done, the private copies are reduced into one — Works because + (and many other operations) are associative 13

Let’s try the version with race conditions § We’re expecting a significant speedup on the 8 -core EWS machines § Not quite 8 -times speedup because of Amdahl’s Law, plus the overhead forking and joining multiple threads § But its actually slower!! Why? ? § Here’s the mental picture that we have: two processors, shared memory total shared variable in memory 14

This mental picture is wrong! § We’ve forgotten about caches! — The memory may be shared, but each processor has its own L 1 cache — As each processor updates total, it bounces between L 1 caches Multiple bouncing slows performance 15

Summary § Performance is of primary concern in some applications — Games, servers, mobile devices, super computers § Many important applications have parallelism — Exploiting it is a good way to speed up programs. § Single Instruction Multiple Data (SIMD) does this at ISA level — Registers hold multiple data items, instruction operate on them — Can achieve factor or 2, 4, 8 speedups on kernels — May require some restructuring of code to expose parallelism § Exploiting core-level parallelism — Watch out for data races! (Makes code incorrect) 16

Cachegrind comparison § I refs: I 1 misses: L 2 i misses: I 1 miss rate: L 2 i miss rate: Version 1 107, 147 1, 412 1, 372 0. 00% § D refs: D 1 misses: L 2 d misses: D 1 miss rate: L 2 d miss rate: 48, 255, 513 352, 425 17, 279 0. 7% 0. 0% § L 2 refs: L 2 misses: L 2 miss rate: 353, 837 18, 651 0. 0% Version 2 1, 324, 684 1, 078 1, 059 0. 08% 0. 07% 446, 009 8, 190 4, 942 1. 8% 1. 1% 9, 268 6, 001 0. 3% 17