New Algorithms for SIMD Alignment Liza Fireman Technion

New Algorithms for SIMD Alignment Liza Fireman - Technion Ayal Zaks – IBM Haifa Research Lab Erez Petrank – Microsoft Research & Technion Fireman, Petrank & Zaks SIMD Alignment Alg's CC 2007

SIMD (Single Instruction Multiple Data) • Support packed vector operations. + = Fireman, Petrank & Zaks SIMD Alignment Alg's 2

SIMD (Single Instruction Multiple Data) • Support packed vector operations. • Widely used with multimedia extensions. – Altivec (IBM, Motorola), SSE (Intel). • Manual programming for SIMD is error prone. • Automatically generating optimized code for SIMD (auto-vectorization or simdization) is challenging, but promising. • One challenge: satisfy the alignment constraint imposed by the hardware. – Altivec: 16 -bytes registers are loaded from 16 -bytes consecutive and aligned memory locations. Fireman, Petrank & Zaks SIMD Alignment Alg's 3

Misaligned Streams Example: for (i = 0; i < 1000; i++) a[i] = b[i+1] + c[i+2]; • The above code requires additional realignment operations. … b[i-1] b[i+1] b[i+2] b[i+3] b[i+4] b[i+5]b[i+6] b[i+7] b[i+1] b[i+2] b[i+3] Fireman, Petrank & Zaks … b[i+4] SIMD Alignment Alg's 4

The SIMD Alignment Problem • Given an expression, execute it with the minimum number of “shifts”. • Requirements: – Input and output operands come with a specified alignment – The inputs to each operation have the same alignment • Usually shifts are executed inside a loop and have a noticeable cost. Fireman, Petrank & Zaks SIMD Alignment Alg's 5

A Graph Abstraction • Represent expressions as graphs (standard. ( • Annotate alignments of inputs and outputs. • Solution provides alignments for the inner vertices (the operations): alignment of the operation inputs. • Mapping of graph solutions to expression solutions (and vice versa) is easy if each array appears in a single alignment only. Fireman, Petrank & Zaks SIMD Alignment Alg's 6

A (Tree) Example b c d 3 1 e 3 2 3 * a[i+3]= + 3 b[i+1]*c[i+3] + d[i+3]*e[i+2] a 3 Fireman, Petrank & Zaks SIMD Alignment Alg's 7

Previous Heuristics • Several simple heuristics have been proposed to solve the alignment problem. • • The Zero-Shift Policy The Eager-Shift Policy The Lazy-Shift Policy The Majority Policy Fireman, Petrank & Zaks SIMD Alignment Alg's 8

Talk Outline • Introduction: SIMD alignment, graph abstraction, heuristics. • Tree expressions: dynamic programming. • Expressions with two alignments: node multiway cuts. • The general case • Measurements • Conclusions Fireman, Petrank & Zaks SIMD Alignment Alg's 9

Two Interesting Special Cases • Single-appearance tree expressions – Each array appears once in the input. • Expressions with only two alignments – Each array appears with only one of the alignments • We present two efficient algorithms that solve the problem optimally for these two cases. Fireman, Petrank & Zaks SIMD Alignment Alg's 10

A Tree Example b c d 3 1 e 3 * 2 * a[i+3]= + b[i+1]*c[i+3] + d[i+3]*e[i+2] a 3 Fireman, Petrank & Zaks SIMD Alignment Alg's 11

Optimal Algorithm for a Tree • Dynamic programming. – Progressive local computations for the global optimum. l Fireman, Petrank & Zaks m SIMD Alignment Alg's j 12

Optimal Algorithm for a Tree • Dynamic programming. – Progressive local computations for the global optimum. 1 2 … i … k l 1 2 Fireman, Petrank & Zaks j m … … i … 1 2 … i … k k k SIMD Alignment Alg's 13

1 2 3 0 ∞ ∞ 1 1 2 3 ∞ ∞ 0 b c 2 3 1 2 1 3 1 ∞ ∞ 0 ∞ d 3 1 1 2 3 0 ∞ e 3 2 * * + 2 1 2 3 3 3 2 1 2 3 2 1 1 a 3 1 Fireman, Petrank & Zaks 2 3 Alg's ∞ Alignment ∞SIMD 2 14

Complexity – Tree Expressions • Traverse the tree nodes, for each possible alignment, do work for each incoming edge. • Overall O(k|V|), where k is the number of possible alignment. Fireman, Petrank & Zaks SIMD Alignment Alg's 15

It Doesn’t Work on DAGs 0 1 2 1 1 * 0 0 1 2 1 1 1 0 + 1 + Fireman, Petrank & Zaks SIMD Alignment Alg's 16

Two-Alignments Expressions • Not necessarily trees. • Only two alignments in the expression. Fireman, Petrank & Zaks SIMD Alignment Alg's 17

for (i = 0; i < 1000; i++) f[i] = (b[i+1]*a[i+1] + c[i+1]*a[i+1]) + (d[i]*a[i+1] + e[i]*a[i+1]) b c 1 1 d a 0 1 1 * e 0 0 * 0 + 1 + 0 + f Fireman, Petrank & Zaks SIMD 0 Alignment Alg's 18

for (i = 0; i < 1000; i++) f[i] = (b[i+1]*a[i+1] + c[i+1]*a[i+1]) + (d[i]*a[i+1] + e[i]*a[i+1]) b c 1 1 d a 0 1 1 * e 0 * 0 + 1 + 0 Only a single shift required here. f Fireman, Petrank & Zaks SIMD 0 Alignment Alg's 19

Node Multi-way cut a S 1 1 b d c 2 2 f e 3 3 S 2 Fireman, Petrank & Zaks 3 SIMD Alignment Alg's 20

Using Node Multiway Cuts a c b 0 0 * 1 • Choosing a node for the cut means: shift after executing it. • To make sure that all inputs of an operation get aligned: link them all to each other ! – Moral graphs d 1 Fireman, Petrank & Zaks SIMD Alignment Alg's 21

S 1 b S 0 c 1 1 d a 0 1 1 * e 0 0 * 0 + 1 + 0 + f Fireman, Petrank & Zaks SIMD 0 Alignment Alg's 22

for (i = 0; i < 1000; i++) f[i] = (b[i+1]*a[i+1] + c[i+1]*a[i+1]) + (d[i]*a[i+1] + e[i]*a[i+1]) b c 1 1 d a 0 1 1 * 0 + 1 + 0 0 * Each cut node implies one shift e Cost: 2 shifts 0 + f Fireman, Petrank & Zaks SIMD 0 Alignment Alg's 23

The Algorithm • • Create the modified graph. Find a 2 -way min-cut via max-flow algorithms. Complexity: min-cut on modified graph O(|V|4). 3 -way and up node-cuts are NP-Hard. • The node-cut and edge-cut problems are different from the SIMD alignment problem. • But relations exist. Derive approximation alg’s for SIMD from approx. alg. for node-cut and edge-cut. (See paper. ) • No NP-Completeness result known for this problem. Fireman, Petrank & Zaks SIMD Alignment Alg's 24

Measurements • Part 1: How much does it cost to shift ? • Part 2: Generate Random graphs and check: OPT/HEU for random graphs – Single-appearance trees – 2 -alignment DAGS • OPT = # shifts used by the optimal solution • HEU = # shifts used by the best heuristic Fireman, Petrank & Zaks SIMD Alignment Alg's 25

Part 1: cost of shift a b c 1 1 d 1 1 Cost of best heuristic: 2. Cost of optimal solution: 1 f e 2 Fireman, Petrank & Zaks 2 SIMD Alignment Alg's 26

Part 1: cost of shift a b c 1 1 d 1 1 Cost of best heuristic: 2. Cost of optimal solution: 1 6% runtime improvement f e 2 Fireman, Petrank & Zaks 2 SIMD Alignment Alg's 27

Random Trees: OPT/HEU Fireman, Petrank & Zaks SIMD Alignment Alg's 28

Random Layered Graphs: OPT/HEY w – width of layered graph d – graph’s depth Fireman, Petrank & Zaks SIMD Alignment Alg's 29

Related Work • [Eichnberg-Wu-O’Brien PLDI’ 04]: – set of alignment heuristics. – Code generation (can use our algorihtms). • [Wu-Eichnberg-Wang CGO 2005] – Runtime alignments • Several compilers (e. g. , GCC, VAST, compilers for SSE) use the zero-shift policy. • [Ren-Wu-Padua PLDI 2006] handle strides > 1. • Much literature on distributing data to processors. Fireman, Petrank & Zaks SIMD Alignment Alg's 30

Summary • The SIMD alignment problem is important. • Previously only heuristics were used • We propose optimal algorithms for: – single-appearance tree expressions – expressions with only two alignments • Guaranteed approximation ratio for the general case. • Measurements show that optimizations are effective. • Future work: is SIMD-Alignment NP-Complete, or can you solve it? – More special cases? Fireman, Petrank & Zaks SIMD Alignment Alg's 31