Idealized Piecewise Linear Branch Prediction Daniel A Jimnez

Idealized Piecewise Linear Branch Prediction Daniel A. Jiménez Department of Computer Science Rutgers University and Departament d'Arquitectura de Computadors Universitat Politècnica de Catalunya

This Talk u Brief introduction to conditional branch prediction u u Introduction to neural branch prediction u u Perceptron predictor Mathematical intuition Some pictures and movies Piecewise Linear Branch Prediction u u u Some motivation, some background The algorithm Why it’s better Idealized Piecewise Linear Branch Prediction u My entry into the championship branch predictor contest 2

Pipelining and Branches Pipelining overlaps instructions to exploit parallelism, allowing the clock rate to be increased. Branches cause bubbles in the pipeline, where some stages are left idle. Instruction fetch Instruction decode Execute Memory access Write back Unresolved branch instruction 3

Branch Prediction A branch predictor allows the processor to speculatively fetch and execute instructions down the predicted path. Instruction fetch Instruction decode Execute Memory access Write back Speculative execution Branch predictors must be highly accurate to avoid mispredictions! 4

Branch Predictors Must Improve u u The cost of a misprediction is proportional to pipeline depth As pipelines deepen, we need more accurate branch predictors u Pentium 4 pipeline has 31 stages! Deeper pipelines allow higher clock rates by decreasing the delay of each pipeline stage u Decreasing misprediction rate from 9% to 4% results in 31% speedup for 32 stage pipeline u Simulations with Simple. Scalar/Alpha 5

Previous Work on Branch Prediction u u The architecture literature is replete with branch prediction papers Most refine two-level adaptive branch prediction [Yeh & Patt 91] u u A 1 st-level table records recent global or per-branch pattern histories A 2 nd-level table learns correlations between histories and outcomes Refinements focus on reducing destructive interference in the tables Some of the better refinements (not an exhaustive list): u gshare [Mc. Farling `93] agree [Sprangle et al. `97] hybrid predictors [Evers et al. `96] u skewed predictors u u [Michaud et al. `93] 6

Conditional Branch Prediction is a Machine Learning Problem u The machine learns to predict conditional branches u So why not apply a machine learning algorithm? u Artificial neural networks u Simple model of neural networks in brain cells u Learn to recognize and classify patterns u We used fast and accurate perceptrons [Rosenblatt `62, Block `62] for dynamic branch prediction [Jiménez & Lin, HPCA 2001] u We were the first to use single-layer perceptrons and to achieve accuracy superior to PHT techniques. Previous work used LVQ and MLP for branch prediction [Vintan & Iridon `99]. 7

Input and Output of the Perceptron u u u The inputs to the perceptron are branch outcome histories u Just like in 2 -level adaptive branch prediction u Can be global or local (per-branch) or both (alloyed) u Conceptually, branch outcomes are represented as u +1, for taken u -1, for not taken The output of the perceptron is u Non-negative, if the branch is predicted taken u Negative, if the branch is predicted not taken Ideally, each static branch is allocated its own perceptron 8

Branch-Predicting Perceptron u u Inputs (x’s) are from branch history and are -1 or +1 n + 1 small integer weights (w’s) learned by on-line training Output (y) is dot product of x’s and w’s; predict taken if y ≥ 0 Training finds correlations between history and outcome 9

Training Algorithm 10

What Do The Weights Mean? u The bias weight, w 0: u u u The correlating weights, w 1 through wn: u u wi is proportional to the probability that the predicted branch agrees with the ith branch in the history The dot product of the w’s and x’s u u u Proportional to the probability that the branch is taken Doesn’t take into account other branches; just like a Smith predictor wi × xi is proportional to the probability that the predicted branch is taken based on the correlation between this branch and the ith branch Sum takes into account all estimated probabilities What’s θ? u Keeps from overtraining; adapt quickly to changing behavior 11

Organization of the Perceptron Predictor u u Keeps a table of m perceptron weights vectors Table is indexed by branch address modulo m [Jiménez & Lin, HPCA 2001] 12

Mathematical Intuition A perceptron defines a hyperplane in n+1 -dimensional space: For instance, in 2 D space we have: This is the equation of a line, the same as 13

Mathematical Intuition continued In 3 D space, we have Or you can think of it as i. e. the equation of a plane in 3 D space This hyperplane forms a decision surface separating predicted taken from predicted not taken histories. This surface intersects the feature space. Is it a linear surface, e. g. a line in 2 D, a plane in 3 D, a cube in 4 D, etc. 14

Example: AND u u u Here is a representation of the AND function White means false, black means true for the output -1 means false, +1 means true for the input -1 AND -1 = false -1 AND +1 = false +1 AND -1 = false +1 AND +1 = true 15

Example: AND continued u A linear decision surface (i. e. a plane in 3 D space) intersecting the feature space (i. e. the 2 D plane where z=0) separates false from true instances 16

Example: AND continued u Watch a perceptron learn the AND function: 17

Example: XOR u Here’s the XOR function: -1 XOR -1 = false -1 XOR +1 = true +1 XOR -1 = true +1 XOR +1 = false Perceptrons cannot learn such linearly inseparable functions 18

My Previous Work on Neural Predictors u u The perceptron predictor uses only pattern history information u The same weights vector is used for every prediction of a static branch u The ith history bit could come from any number of static branches u So the ith correlating weight is aliased among many branches The newer path-based neural predictor uses path information u The ith correlating weight is selected using the ith branch address u This allows the predictor to be pipelined, mitigating latency u This strategy improves accuracy because of path information u But there is now even more aliasing since the ith weight could be used to predict many different branches 19

Piecewise Linear Branch Prediction u Generalization of perceptron and path-based neural predictors u Ideally, there is a weight giving the correlation between each u u Static branch b, and u Each pair of branch and history position (i. e. i) in b’s history b might have 1000 s of correlating weights or just a few u u Depends on the number of static branches in b’s history First, I’ll show a “practical version” 20

The Algorithm: Parameters and Variables u GHL – the global history length u GHR – a global history shift register u GA – a global array of previous branch addresses u W – an n × m × (GHL + 1) array of small integers 21

The Algorithm: Making a Prediction Weights are selected based on the current branch and the ith most recent branch 22

The Algorithm: Training 23

Why It’s Better u Forms a piecewise linear decision surface u u Each piece determined by the path to the predicted branch Can solve more problems than perceptron Perceptron decision surface for XOR Piecewise linear decision surface for XOR doesn’t classify all inputs correctly classifies all inputs correctly 24

Learning XOR u From a program that computes XOR using if statements perceptron prediction piecewise linear prediction 25

A Generalization of Neural Predictors u When m = 1, the algorithm is exactly the perceptron predictor u u W[n, 1, h+1] holds n weights vectors When n = 1, the algorithm is path-based neural predictor u u W[1, m, h+1] holds m weights vectors Can be pipelined to reduce latency u The design space in between contains more accurate predictors u If n is small, predictor can still be pipelined to reduce latency 26

Generalization Continued Perceptron and pathbased are the least accurate extremes of piecewise linear branch prediction! 27

Idealized Piecewise Linear Branch Prediction u Presented at CBP workshop at MICRO 2004 u Hardware budget limited to 64 K + 256 bits u No other limitations u Get rid of n and m u Allow 1 st and 2 nd dimensions of W to be unlimited u Now branches cannot alias one another; accuracy much better u One small problem: unlimited amount of storage required u How to squeeze this into 65, 792 bits for the contest? 28

Hashing u u 3 indices of W : i, j, & k, index arbitrary numbers of weights Hash them into 0. . N-1 weights in an array of size N Collisions will cause aliasing, but more uniformly distributed Hash function uses three primes H 1 H 2 and H 3: 29

More Tricks u u u Weights are 7 bits, elements of GA are 8 bits Separate arrays for bias weights and correlating weights Using global and per-branch history u u u Slightly bias the predictor toward not taken Dynamically adjust history length u u Based on an estimate of the number of static branches Extra weights u u u An array of per-branch histories is kept, alloyed with global history Extra bias weights for each branch Extra correlating weights for more recent history bits Inverted bias weights that track the opposite of the branch bias 30

Parameters to the Algorithm #define NUM_WEIGHTS 8590 #define NUM_BIASES 599 #define INIT_GLOBAL_HISTORY_LENGTH 30 #define HIGH_GLOBAL_HISTORY_LENGTH 48 #define LOW_GLOBAL_HISTORY_LENGTH 18 #define INIT_LOCAL_HISTORY_LENGTH 4 #define HIGH_LOCAL_HISTORY_LENGTH 16 #define LOW_LOCAL_HISTORY_LENGTH 1 #define EXTRA_BIAS_LENGTH 6 #define HIGH_EXTRA_BIAS_LENGTH 2 #define LOW_EXTRA_BIAS_LENGTH 7 #define EXTRA_HISTORY_LENGTH 5 #define HIGH_EXTRA_HISTORY_LENGTH 7 #define LOW_EXTRA_HISTORY_LENGTH 4 #define INVERTED_BIAS_LENGTH 8 #define HIGH_INVERTED_BIAS_LENGTH 4 #define LOW_INVERTED_BIAS_LENGTH 9 #define NUM_HISTORIES 55 #define WEIGHT_WIDTH 7 #define MAX_WEIGHT 63 #define MIN_WEIGHT -64 #define INIT_THETA_UPPER 70 #define INIT_THETA_LOWER -70 #define HIGH_THETA_UPPER 139 #define HIGH_THETA_LOWER -136 #define LOW_THETA_UPPER 50 #define LOW_THETA_LOWER -46 #define HASH_PRIME_1 511387 U #define HASH_PRIME_2 660509 U #define HASH_PRIME_3 1289381 U #define TAKEN_THRESHOLD 3 All determined empirically with an ad hoc approach 31

Per-Benchmark Accuracy u u I used several highly accurate predictors to compete with my predictor I measured the potential of my technique using an unlimited hardware budget 32

Scores for the 6 Finalists (out of 18 entries) scores are in average MPKI (mispredicts per 1000 insts) (corrected) over a suite of 20 traces from Intel 1. Hongliang Gao, Huiyang Zhou 2. 574 2. André Seznec 2. 627 3. Gabriel Loh 2. 700 4. Daniel A. Jiménez 2. 742 5. Pierre Michaud 2. 777 6. Veerle Desmet et al. 2. 807 5 of the 6 finalists used ideas from the perceptron predictor in their entries 33

References u Jiménez and Lin, HPCA 2001 (perceptron predictor) u Jiménez and Lin, TOCS 2002 (global/local perceptron) u Jiménez, MICRO 2003 (path-based neural predictor) u Jiménez, ISCA 2005 (piecewise linear branch prediction) u Juan, Sanjeevan, Navarro, SIGARCH Comp. News, 1998 (dynamic history length fitting) u Skadron, Martonosi, Clark, PACT 2000 (alloyed history) 34

The End 35

Extra Slides u Following this slide are some slides cut from the talk to fit within time constraints. 36

Program to Compute XOR int f () { int a, b, x, i, s = 0; for (i=0; i<100; i++) { a = rand () % 2; b = rand () % 2; if (a) { if (b) else } else { if (b) else } } return s; x = 0; x = 1; x = 0; } if (x) s++; /* this is the branch */ 37

Example: XOR continued u Watch a perceptron try to learn XOR 38