Graduate Computer Architecture I Lecture 3 Branch Prediction

Graduate Computer Architecture I Lecture 3: Branch Prediction Young Cho

Cycles Per Instructions “Average Cycles per Instruction” CPI = (CPU Time * Clock Rate) / Instruction Count = Cycles / Instruction Count “Instruction Frequency” 2 - CSE/ESE 560 M – Graduate Computer Architecture I

Typical Load/Store Processor IF/ID PC Control ID/EX Register File EX/MEM MEM/WB ALU Data Memory Instruction Memory 3 - CSE/ESE 560 M – Graduate Computer Architecture I

Pipelining Laundry 30 minutes 35 25 minutes 3 X Increase in Productivity!!! With large number of sets, the each load takes average of ~35 min to wash Three sets of Clean Clothes in 2 hours 40 minutes 4 - CSE/ESE 560 M – Graduate Computer Architecture I

Introducing Problems • Hazards prevent next instruction from executing during its designated clock cycle – Structural hazards: HW cannot support this combination of instructions (single person to dry and iron clothes simultaneously) – Data hazards: Instruction depends on result of prior instruction still in the pipeline (missing sock – needs both before putting them away) – Control hazards: Caused by delay between the fetching of instructions and decisions about changes in control flow (Er…branch & jump) 5 - CSE/ESE 560 M – Graduate Computer Architecture I

Data Hazards • Read After Write (RAW) – Instr 2 tries to read operand before Instr 1 writes it – Caused by a “Dependence” in compiler term • Write After Read (WAR) – Instr 2 writes operand before Instr 1 reads it – Called an “anti-dependence” in compiler term • Write After Write (WAW) – Instr 2 writes operand before Instr 1 writes it – “Output dependence” in compiler term • WAR and WAW in more complex systems 6 - CSE/ESE 560 M – Graduate Computer Architecture I

r 6, r 1, r 7 22: add r 8, r 1, r 9 Reg DMem Ifetch Reg ALU 18: or Ifetch DMem ALU 14: and r 2, r 3, r 5 Reg ALU Ifetch ALU 10: beq r 1, r 3, 36 ALU Branch Hazard (Control) 36: xor r 10, r 11 3 instructions are in the pipeline before new instruction can be fetched. 7 - CSE/ESE 560 M – Graduate Computer Architecture I Reg Reg DMem

Branch Hazard Alternatives • Stall until branch direction is clear • Predict Branch Not Taken – – – Execute successor instructions in sequence “Squash” instructions in pipeline if branch actually taken Advantage of late pipeline state update 47% DLX branches not taken on average PC+4 already calculated, so use it to get next instr • Predict Branch Taken – 53% DLX branches taken on average – DLX still incurs 1 cycle branch penalty – Other machines: branch target known before outcome 8 - CSE/ESE 560 M – Graduate Computer Architecture I

Branch Hazard Alternatives • Delayed Branch – Define branch to take place AFTER a following instruction (Fill in Branch Delay Slot) branch instruction sequential successor 1 sequential successor 2. . . . sequential successorn branch target if taken Branch delay of length n – 1 slot delay allows proper decision and branch target address in 5 stage pipeline 9 - CSE/ESE 560 M – Graduate Computer Architecture I

Evaluating Branch Alternatives Scheduling scheme Branch CPI penalty speedup v. unpipelined Stall pipeline Predict taken Predict not taken Delayed branch 3 1 1 0. 5 3. 5 4. 4 4. 5 4. 6 1. 42 1. 14 1. 09 1. 07 speedup v. stall 1. 0 1. 26 1. 29 1. 31 Conditional & Unconditional = 14%, 65% change PC 10 - CSE/ESE 560 M – Graduate Computer Architecture I

Solution to Hazards • Structural Hazards – Delaying HW Dependent Instruction – Increase Resources (i. e. dual port memory) • Data Hazards – Data Forwarding – Software Scheduling • Control Hazards – Pipeline Stalling – Predict and Flush – Fill Delay Slots with Previous Instructions 11 - CSE/ESE 560 M – Graduate Computer Architecture I

Administrative • Literature Survey – One Q&A per Literature – Q&A should show that you read the paper • Changes in Schedule – Need to be out of town on Oct 4 th (Tuesday) – Quiz 2 moved up 1 lecture • Tool and VHDL help 12 - CSE/ESE 560 M – Graduate Computer Architecture I

Typical Pipeline • Example: MIPS R 4000 integer unit ex FP/int Multiply IF ID m 1 m 2 m 3 m 4 m 5 m 6 FP adder a 1 a 2 a 3 a 4 FP/int divider Div (lat = 25, Init inv=25) 13 - CSE/ESE 560 M – Graduate Computer Architecture I m 7 MEM WB

Prediction • Easy to fetch multiple (consecutive) instructions per cycle – Essentially speculating on sequential flow • Jump: unconditional change of control flow – Always taken • Branch: conditional change of control flow – Taken typically ~50% of the time in applications • Backward: 30% of the Branch 80% taken = ~24% • Forward: 70% of the Branch 40% taken = ~28% 14 - CSE/ESE 560 M – Graduate Computer Architecture I

Current Ideas • Reactive – Adapt Current Action based on the Past – TCP windows – URL completion, . . . • Proactive – Anticipate Future Action based on the Past – Branch prediction – Long Cache block – Tracing 15 - CSE/ESE 560 M – Graduate Computer Architecture I

Branch Prediction Schemes • Static Branch Prediction • Dynamic Branch Prediction – 1 -bit Branch-Prediction Buffer – 2 -bit Branch-Prediction Buffer – Correlating Branch Prediction Buffer – Tournament Branch Predictor • Branch Target Buffer • Integrated Instruction Fetch Units • Return Address Predictors 16 - CSE/ESE 560 M – Graduate Computer Architecture I

Static Branch Prediction • Execution profiling – Very accurate if Actually take time to Profile – Incovenient • Heuristics based on nesting and coding – Simple heuristics are very inaccurate • Programmer supplied hints. . . – Inconvenient and potentially inaccurate 17 - CSE/ESE 560 M – Graduate Computer Architecture I

Dynamic Branch Prediction • Performance = ƒ(accuracy, cost of mis-prediction) • 1 -bit Branch History Table – Bitmap for Lower bits of PC address – Says whether or not branch taken last time – If Inst is Branch, predict and update the table • Problem – 1 -bit BHT will cause 2 mis-predictions for Loops • First time through the loop, it predicts exit instead loop • End of loop case, it predicts loops instead of exit – Avg is 9 iterations before exit • Only 80% accuracy even if loop 90% of the time 18 - CSE/ESE 560 M – Graduate Computer Architecture I

N-bit Dynamic Branch Prediction • N-bit scheme where change prediction only if get misprediction N-times: T Predict Taken NT Predict Taken T T NT NT Predict Not Taken T NT 2 -bit Scheme: Saturates the prediction up to 2 times 19 - CSE/ESE 560 M – Graduate Computer Architecture I

Correlating Branches • (2, 2) predictor Branch address (4 bits) – 2 -bit global: indicates the behavior of the last two branches – 2 -bit local (2 -bit Dynamic Branch Prediction) • Branch History Table – Global branch history is used to choose one of four history bitmap table – Predicts the branch behavior then updates only the selected bitmap table 20 - CSE/ESE 560 M – Graduate Computer Architecture I Prediction 2 -bit recent global branch history (01 = not taken then taken)

Accuracy of Different Schemes 20% Frequency of Mispredictions 18% 4096 Entries 2 -bit BHT Unlimited Entries 2 -bit BHT 1024 Entries (2, 2) BHT 16% 14% 12% 18% 11% 10% 8% 6% 6% 5% 5% 4% 4% 2% 1% 1% 0% 0% nasa 7 matrix 300 tomcatv doducd 21 - CSE/ESE 560 M – Graduate Computer Architecture I spice fpppp gcc espresso eqntott li

BHT Accuracy • Mispredict because either: – Wrong guess for the branch – Wrong Index for the branch • 4096 entry table – programs vary from 1% misprediction (nasa 7, tomcatv) to 18% (eqntott), with spice at 9% and gcc at 12% • For SPEC 92 – 4096 about as good as infinite table 22 - CSE/ESE 560 M – Graduate Computer Architecture I

Tournament Branch Predictors • Correlating Predictor – 2 -bit predictor failed on important branches – Better results by also using global information • Tournament Predictors – 1 Predictor based on global information – 1 Predictor based on local information – Use the predictor that guesses better addr Predictor A 23 - CSE/ESE 560 M – Graduate Computer Architecture I Predictor B

Alpha 21264 • • 4 K 2 -bit counters to choose from among a global predictor and a local predictor Global predictor also has 4 K entries and is indexed by the history of the last 12 branches; each entry in the global predictor is a standard 2 -bit predictor – 12 -bit pattern: ith bit 0 => ith prior branch not taken; ith bit 1 => ith prior branch taken; • Local predictor consists of a 2 -level predictor: • Total size: 4 K*2 + 1 K*10 + 1 K*3 = 29 K bits! (~180, 000 transistors) – Top level a local history table consisting of 1024 10 -bit entries; each 10 -bit entry corresponds to the most recent 10 branch outcomes for the entry. 10 -bit history allows patterns 10 branches to be discovered and predicted. – Next level Selected entry from the local history table is used to index a table of 1 K entries consisting a 3 -bit saturating counters, which provide the local prediction 24 - CSE/ESE 560 M – Graduate Computer Architecture I

Branch Prediction Accuracy 99% 100% tomcatv 95% doduc 84% fpppp 86% 82% li 77% 97% 88% 98% 86% 82% espresso gcc 70% 0% 20% 40% 25 - CSE/ESE 560 M – Graduate Computer Architecture I 60% 80% 98% 96% 88% 94% 100% Profile-based 2 -bit dynmic Tournament

Accuracy versus Size 26 - CSE/ESE 560 M – Graduate Computer Architecture I

Branch Target Buffer • Branch Target Buffer (BTB): Address of branch index to get prediction AND branch address (if taken) – Note: must check for branch match now, since can’t use wrong branch address Branch PC Predicted PC PC of instruction FETCH =? No: branch not predicted, proceed normally (Next PC = PC+4) 27 - CSE/ESE 560 M – Graduate Computer Architecture I Yes: instruction is branch and use predicted PC as next PC Extra prediction state bits

Predicated Execution • Built in Hardware Support – Bit for predicated instruction execution – Both paths are in the code – Execution based on the result of the condition • No Branch Prediction is Required – Instructions not selected are ignored – Sort of inserting Nop 28 - CSE/ESE 560 M – Graduate Computer Architecture I

Zero Cycle Jump • What really has to be done at runtime? – Once an instruction has been detected as a jump or JAL, we might recode it in the internal cache. – Very limited form of dynamic compilation? • Use of “Pre-decoded” instruction cache – Called “branch folding” in the Bell-Labs CRISP processor. – Original CRISP cache had two addresses and could thus fold a complete branch into the previous instruction – Notice that JAL introduces a structural hazard on write Internal Cache state: and addi sub jal subi r 3, r 1, r 5 r 2, r 3, #4 r 4, r 2, r 1 doit r 1, #1 A: and r 3, r 1, r 5 N A+4 addi r 2, r 3, #4 N A+8 sub r 4, r 2, r 1 L doit --- -- --- r 1, #1 N A+20 subi 29 - CSE/ESE 560 M – Graduate Computer Architecture I

Dynamic Branch Prediction Summary • Prediction becoming important part of scalar execution • Branch History Table – 2 bits for loop accuracy • Correlation – Recently executed branches correlated with next branch. – Either different branches – Or different executions of same branches • Tournament Predictor – More resources to competitive solutions and pick between them • Branch Target Buffer – Branch address & prediction • Predicated Execution – No need for Prediction – Hardware Support needed 30 - CSE/ESE 560 M – Graduate Computer Architecture I