COMPUTER ARCHITECTURE CS 6354 Out of Order Execution

COMPUTER ARCHITECTURE CS 6354 Out of Order Execution Samira Khan University of Virginia Feb 16, 2016 The content and concept of this course are adapted from CMU ECE 740

AGENDA • Logistics • Review from last lecture • Out of order execution • Issues in Out of Order Execution 2

LOGISTICS • Course feedback form: Feb 18 • Literature survey: Feb 15 • Project proposal: Feb 18, will have 4 late days • Use the latex template • Follow the format of the sample proposals • Each group will prepare one document, but upload the file individually via Collab 3

RECAP OF LAST LECTURE • Maintaining Speculative Memory State (Ld/St Ordering) • Out of Order Execution (Dynamic Scheduling) – Link Dependent Instructions: Renaming – Buffer Instructions: Reservation Stations – Track Readiness of Source Values: Tag (and Value) Broadcast – Schedule/Dispatch: Wakeup and Select • Tomasulo’s Algorithm • Oo. O Execution Exercise with Code Example: Cycle by Cycle 4

IN-ORDER VS. OUT-OF-ORDER DISPATCH • In order dispatch + precise exceptions: F D E E E STALL F R W E R W D E R W F D E E STALL E R W E R IMUL ADD R 3 R 1, R 2 R 3, R 1 R 6, R 7 R 5 R 6, R 8 R 7 R 3, R 5 W • Out-of-order dispatch + precise exceptions: F D E F D F E E E WAIT D E R F D E F D R W E R W W E E WAIT E R W • 16 vs. 12 cycles 5

ENABLING OOO EXECUTION, REVISITED 1. Link the consumer of a value to the producer – Register renaming: Associate a “tag” with each data value 2. Buffer instructions until they are ready – Insert instruction into reservation stations after renaming 3. Keep track of readiness of source values of an instruction – Broadcast the “tag” when the value is produced – Instructions compare their “source tags” to the broadcast tag if match, source value becomes ready 4. When all source values of an instruction are ready, dispatch the instruction to functional unit (FU) – Wakeup and select/schedule the instruction 6

SUMMARY OF OOO EXECUTION CONCEPTS • Register renaming eliminates false dependencies, enables linking of producer to consumers • Buffering enables the pipeline to move for independent ops • Tag broadcast enables communication (of readiness of produced value) between instructions • Wakeup and select enables out-of-order dispatch 7

SOME QUESTIONS • What is needed in hardware to perform tag broadcast and value capture? make a value valid wake up an instruction • Does the tag have to be the ID of the Reservation Station Entry? • What can potentially become the critical path? – Tag broadcast value capture instruction wake up • How can you reduce the potential critical paths? 8

AN EXERCISE, WITH PRECISE EXCEPTIONS MUL ADD ADD MUL ADD R 3 R 1, R 2 R 5 R 3, R 4 R 7 R 2, R 6 R 10 R 8, R 9 R 11 R 7, R 10 R 5, R 11 F D E R W • Assume ADD (4 cycle execute), MUL (6 cycle execute) • Assume one adder and one multiplier • How many cycles – in an in-order-dispatch pipelined machine with reorder buffer (no forwarding and full forwarding) – in an out-of-order dispatch pipelined machine with reorder buffer (full forwarding) 9

OUT-OF-ORDER EXECUTION WITH PRECISE EXCEPTIONS • Idea: Use a reorder buffer to reorder instructions before committing them to architectural state • An instruction updates the register alias table (essentially a future file) when it completes execution • An instruction updates the architectural register file when it is the oldest in the machine and has completed execution 10

OUT-OF-ORDER EXECUTION WITH PRECISE EXCEPTIONS TAG and VALUE Broadcast Bus F D S C H E D U L E Integer add E E Integer mul E E E E E FP mul . . . R E O R D E R W Load/store in order out of order • Hump 1: Reservation stations (scheduling window) • Hump 2: Reordering (reorder buffer, aka instruction window or active window) in order 11

MODERN OOO EXECUTION W/ PRECISE EXCEPTIONS • Most modern processors use – Reorder buffer supports in-order retirement of instructions – A single register file stores registers (speculative and architectural) – INT and FP are still separate – Future register map used for renaming – Architectural register map used for state recovery 12

AN EXAMPLE FROM MODERN PROCESSORS Boggs et al. , “The Microarchitecture of the Pentium 4 Processor, ” Intel Technology Journal, 2001. 13

OOO EXECUTION: RESTRICTED DATAFLOW • An out-of-order engine dynamically builds the dataflow graph of a piece of the program – which piece? • The dataflow graph is limited to the instruction window – Instruction window: all decoded but not yet retired instructions • Can we do it for the whole program? • Why would we like to? • In other words, how can we have a large instruction window? • Can we do it efficiently with Tomasulo’s algorithm? 14

DATAFLOW GRAPH FOR OUR EXAMPLE MUL ADD ADD MUL ADD R 3 R 1, R 2 R 5 R 3, R 4 R 7 R 2, R 6 R 10 R 8, R 9 R 11 R 7, R 10 R 5, R 11 15

CYCLE 7 1 2 3 4 5 6 7 8 9 1 1 1 1 1 2 0 1 2 3 4 5 6 7 8 9 0 MUL R 3 R 1, R 2 F D E E E ADD R 5 R 3, R 4 F D _ _ ADD R 7 R 2, R 6 F D E E E F D E E ADD R 10 R 8, R 9 MUL R 11 R 7, R 10 F D _ ADD R 5, R 11 TAG VAL V R 1 1 1 R 2 2 1 R 3 X R 4 R 5 4 d R 6 R 7 0 1 0 6 b 1 0 R 8 8 1 R 9 9 1 R 10 c 0 R 11 y 0 Register Alias Table F D Source 1 a b c d TAG VAL Source 1 Source 2 V TAG VAL V 0 4 1 2 1 6 1 8 1 9 1 X a 0 y 0 TAG x y z t VAL V TAG VAL V 1 1 2 1 b 0 c 0 * + Tag Source 2 Val Tag Val 16

CYCLE 7 Source 1 MUL R 3 R 1, R 2 a b ADD R 5 R 3, R 4 c ADD R 7 R 2, R 6 d ADD R 10 R 8, R 9 MUL R 11 R 7, R 10 ADD R 5, R 11 TAG VAL V R 1 1 1 R 2 2 1 R 3 X R 4 R 5 4 d R 6 R 7 0 TAG VAL V 0 4 1 2 1 6 1 8 1 9 1 X a Source 1 Source 2 0 y 0 TAG x y z t b Source 2 VAL V TAG VAL V 1 1 2 1 0 c 0 R 1 R 2 * X (R 3) 1 0 6 b 1 0 R 8 8 1 R 9 9 1 R 10 c 0 R 11 y 0 Register Alias Table 17

CYCLE 7 Source 1 MUL R 3 R 1, R 2 a b ADD R 5 R 3, R 4 c ADD R 7 R 2, R 6 d ADD R 10 R 8, R 9 MUL R 11 R 7, R 10 ADD R 5, R 11 TAG VAL V R 1 1 1 R 2 2 1 R 3 X R 4 R 5 4 d R 6 R 7 0 TAG VAL V 0 4 1 2 1 6 1 8 1 9 1 X a Source 1 Source 2 0 y 0 TAG x y z t b R 1 R 2 R 6 * + X (R 3) Source 2 VAL V TAG VAL V 1 1 2 1 0 c 0 b (R 7) 1 0 6 b 1 0 R 8 8 1 R 9 9 1 R 10 c 0 R 11 y 0 Register Alias Table 18

CYCLE 7 Source 1 MUL R 3 R 1, R 2 a b ADD R 5 R 3, R 4 c ADD R 7 R 2, R 6 d ADD R 10 R 8, R 9 MUL R 11 R 7, R 10 ADD R 5, R 11 TAG VAL V R 1 1 1 R 2 2 1 R 3 X R 4 R 5 4 d R 6 R 7 0 TAG VAL V 0 4 1 2 1 6 1 8 1 9 1 X a Source 1 Source 2 0 y 0 TAG x y z t b Source 2 VAL V TAG VAL V 1 1 2 1 0 c 0 R 1 R 2 R 6 R 8 R 9 * + + X (R 3) b (R 7) c (R 10) 1 0 6 b 1 0 R 8 8 1 R 9 9 1 R 10 c 0 R 11 y 0 Register Alias Table 19

CYCLE 7 Source 1 MUL R 3 R 1, R 2 a b ADD R 5 R 3, R 4 c ADD R 7 R 2, R 6 d ADD R 10 R 8, R 9 MUL R 11 R 7, R 10 ADD R 5, R 11 TAG VAL V R 1 1 1 R 2 2 1 R 3 X R 4 R 5 4 d R 6 R 7 0 1 0 6 b 1 0 R 8 8 1 R 9 9 1 R 10 c 0 R 11 y 0 Register Alias Table TAG VAL V 0 4 1 2 1 6 1 8 1 9 1 X a Source 1 Source 2 0 y 0 TAG x y z t b Source 2 VAL V TAG VAL V 1 1 2 1 0 c 0 R 1 R 2 R 6 R 8 R 9 * + + X (R 3) b (R 7) X c (R 10) R 4 + a (R 5) 20

CYCLE 7 Source 1 MUL R 3 R 1, R 2 a b ADD R 5 R 3, R 4 c ADD R 7 R 2, R 6 d ADD R 10 R 8, R 9 MUL R 11 R 7, R 10 ADD R 5, R 11 TAG VAL V R 1 1 1 R 2 2 1 R 3 X R 4 R 5 4 d R 6 R 7 0 1 0 6 b 1 0 R 8 8 1 R 9 9 1 R 10 c 0 R 11 y 0 Register Alias Table TAG VAL Source 1 Source 2 V TAG VAL V 0 4 1 2 1 6 1 8 1 9 1 X a 0 y 0 TAG x y z t Source 2 VAL V TAG VAL V 1 1 2 1 b 0 c 0 R 1 R 2 R 6 R 8 R 9 * + + X (R 3) R 5 + a (R 5) b (R 7) b c (R 10) c * y (R 11) 21

CYCLE 7 Source 1 MUL R 3 R 1, R 2 a b ADD R 5 R 3, R 4 c ADD R 7 R 2, R 6 d ADD R 10 R 8, R 9 MUL R 11 R 7, R 10 ADD R 5, R 11 TAG VAL V R 1 1 1 R 2 2 1 R 3 X R 4 R 5 4 d R 6 R 7 0 1 0 6 b 1 0 R 8 8 1 R 9 9 1 R 10 c 0 R 11 y 0 Register Alias Table TAG VAL Source 1 Source 2 V TAG VAL V 0 4 1 2 1 6 1 8 1 9 1 X a 0 y 0 TAG x y z t Source 2 VAL V TAG VAL V 1 1 2 1 b 0 c 0 R 1 R 2 R 6 R 8 R 9 * + + X (R 3) R 5 b (R 7) * + a (R 5) c (R 10) y (R 11) a y + d (R 5) 22

CYCLE 7 Source 1 MUL R 3 R 1, R 2 a b ADD R 5 R 3, R 4 c ADD R 7 R 2, R 6 d ADD R 10 R 8, R 9 MUL R 11 R 7, R 10 ADD R 5, R 11 TAG VAL V R 1 1 1 R 2 2 1 R 3 X R 4 R 5 4 d R 6 R 7 0 1 0 6 b 1 0 R 8 8 1 R 9 9 1 R 10 c 0 R 11 y 0 Register Alias Table TAG VAL Source 1 Source 2 V TAG VAL V 0 4 1 2 1 6 1 8 1 9 1 X a 0 y 0 TAG x y z t Source 2 VAL V TAG VAL V 1 1 2 1 b 0 c 0 R 1 R 2 R 6 R 8 R 9 * + + X (R 3) R 5 b (R 7) + a (R 5) * c (R 10) y (R 11) + d (R 5) 23

QUESTIONS TO PONDER • Why is Oo. O execution beneficial? – What if all operations take single cycle? – Latency tolerance: Oo. O execution tolerates the latency of multi-cycle operations by executing independent operations concurrently • What if an instruction takes 500 cycles? – How large of an instruction window do we need to continue decoding? – How many cycles of latency can Oo. O tolerate? – What limits the latency tolerance scalability of Tomasulo’s algorithm? • Active/instruction window size: determined by both scheduling window and reorder buffer size 24

REGISTERS VERSUS MEMORY, REVISITED • So far, we considered register based value communication between instructions • What about memory? • What are the fundamental differences between registers and memory? – Register dependences known statically – memory dependences determined dynamically – Register state is small – memory state is large – Register state is not visible to other threads/processors – memory state is shared between threads/processors (in a shared memory multiprocessor) 25

MEMORY DEPENDENCE HANDLING (I) • Need to obey memory dependences in an out-of-order machine – and need to do so while providing high performance • Observation and Problem: Memory address is not known until a load/store executes • Corollary 1: Renaming memory addresses is difficult • Corollary 2: Determining dependence or independence of loads/stores need to be handled after their (partial) execution • Corollary 3: When a load/store has its address ready, there may be younger/older loads/stores with undetermined addresses in the machine 26

MEMORY DEPENDENCE HANDLING (II) • When do you schedule a load instruction in an OOO engine? – Problem: A younger load can have its address ready before an older store’s address is known – Known as the memory disambiguation problem or the unknown address problem • Approaches – Conservative: Stall the load until all previous stores have computed their addresses (or even retired from the machine) – Aggressive: Assume load is independent of unknown-address stores and schedule the load right away – Intelligent: Predict (with a more sophisticated predictor) if the load is dependent on the/any unknown address store 27

HANDLING OF STORE-LOAD DEPENDENCES • A load’s dependence status is not known until all previous store addresses are available. • How does the OOO engine detect dependence of a load instruction on a previous store? – Option 1: Wait until all previous stores committed (no need to check for address match) – Option 2: Keep a list of pending stores in a store buffer and check whether load address matches a previous store address • How does the OOO engine treat the scheduling of a load instruction wrt previous stores? – Option 1: Assume load dependent on all previous stores – Option 2: Assume load independent of all previous stores – Option 3: Predict the dependence of a load on an outstanding store 28

MEMORY DISAMBIGUATION (I) • Option 1: Assume load dependent on all previous stores + No need for recovery -- Too conservative: delays independent loads unnecessarily • Option 2: Assume load independent of all previous stores + Simple and can be common case: no delay for independent loads -- Requires recovery and re-execution of load and dependents on misprediction • Option 3: Predict the dependence of a load on an outstanding store + More accurate. Load store dependencies persist over time -- Still requires recovery/re-execution on misprediction – Alpha 21264 : Initially assume load independent, delay loads found to be dependent – Moshovos et al. , “Dynamic speculation and synchronization of data dependences, ” ISCA 1997. – Chrysos and Emer, “Memory Dependence Prediction Using Store Sets, ” ISCA 1998. 29

MEMORY DISAMBIGUATION (II) n n n Chrysos and Emer, “Memory Dependence Prediction Using Store Sets, ” ISCA 1998. Predicting store-load dependencies important for performance Simple predictors (based on past history) can achieve most of the potential performance 30

DATA FORWARDING BETWEEN STS AND LDS • We cannot update memory out of program order Need to buffer all store and load instructions in instruction window • Even if we know all addresses of past stores when we generate the address of a load, two questions still remain: 1. How do we check whether or not it is dependent on a store 2. How do we forward data to the load if it is dependent on a store • Modern processors use a LQ (load queue) and an SQ for this – Can be combined or separate between loads and stores – A load searches the SQ after it computes its address. Why? – A store searches the LQ after it computes its address. Why? 31

CENTRALIZED VS. DISTRIBUTED RESERVATION STATIONS • Centralized (monolithic): + Reservation stations not statically partitioned (can adapt to changes in instruction mix, e. g. , 100% adds) -- All entries need to have all fields even though some fields might not be needed for some instructions (e. g. branches, stores, etc) -- Number of ports = issue width -- More complex control and routing logic • Distributed: + Each RS can be specialized to the functional unit it serves (more area efficient) + Number of ports can be 1 (RS per functional unit) + Simpler control and routing logic -- Other RS’s cannot be used if one functional unit’s RS is full (static partitioning) 32

ISSUES IN SCHEDULING LOGIC (I) • What if multiple instructions become ready in the same cycle? – Why does this happen? • An instruction with multiple dependents • Multiple instructions can complete in the same cycle • Which one to schedule? – Oldest – Random – Most dependents first – Most critical first (longest latency dependency chain) • Does not matter for performance unless the active window is very large – Butler and Patt, “An Investigation of the Performance of Various Dynamic Scheduling Techniques, ” MICRO 1992. 33

ISSUES IN SCHEDULING LOGIC (II) • When to schedule the dependents of a multi-cycle execute instruction? – One cycle before the completion of the instruction – Example: IMUL, Pentium 4, 3 -cycle ADD • When to schedule the dependents of a variablelatency execute instruction? – A load can hit or miss in the data cache – Option 1: Schedule dependents assuming load will hit – Option 2: Schedule dependents assuming load will miss 34

SCHEDULING OF LOAD DEPENDENTS • Assume load will hit + No delay for dependents (load hit is the common case) -- Need to squash and re-schedule if load actually misses • Assume load will miss (i. e. schedule when load data ready) + No need to re-schedule (simpler logic) -- Significant delay for load dependents if load hits • Predict load hit/miss + No delay for dependents on accurate prediction -- Need to predict and re-schedule on misprediction • Yoaz et al. , “Speculation Techniques for Improving Load Related Instruction Scheduling, ” ISCA 1999. 35

COMPUTER ARCHITECTURE CS 6354 Out of Order Execution Samira Khan University of Virginia Feb 16, 2016 The content and concept of this course are adapted from CMU ECE 740

These slides were not covered in the class 37

WHAT TO DO WITH DEPENDENTS ON A LOAD MISS? (I) • A load miss can take hundreds of cycles • If there are many dependent instructions on a load miss, these can clog the scheduling window • Independent instructions cannot be allocated reservation stations and scheduling can stall 38

SMALL WINDOWS: FULL-WINDOW STALLS • When a long-latency instruction is not complete, it blocks retirement. • Incoming instructions fill the instruction window. • Once the window is full, processor cannot place new instructions into the window. – This is called a full-window stall. • A full-window stall prevents the processor from making progress in the execution of the program. 39

SMALL WINDOWS: FULL-WINDOW STALLS 8 -entry instruction window: Oldest LOAD R 1 mem[R 5] L 2 Miss! Takes 100 s of cycles. BEQ R 1, R 0, target ADD R 2, 8 LOAD R 3 mem[R 2] MUL R 4, R 3 ADD R 4, R 5 Independent of the L 2 miss, executed out of program order, but cannot be retired. STOR mem[R 2] R 4 ADD R 2, 64 LOAD R 3 mem[R 2] Younger instructions cannot be executed because there is no space in the instruction window. The processor stalls until the L 2 Miss is serviced. • L 2 cache misses are responsible for most full-window stalls. 40

IMPACT OF L 2 CACHE MISSES L 2 Misses 512 KB L 2 cache, 500 -cycle DRAM latency, aggressive stream-based prefetcher Data averaged over 147 memory-intensive benchmarks on a high-end x 86 processor model 41

IMPACT OF L 2 CACHE MISSES L 2 Misses 500 -cycle DRAM latency, aggressive stream-based prefetcher Data averaged over 147 memory-intensive benchmarks on a high-end x 86 processor model 42

WHAT TO DO WITH DEPENDENTS ON A LOAD MISS? (I) • Idea: Move miss-dependent instructions into a separate buffer – Example: Pentium 4’s “scheduling loops” – Lebeck et al. , “A Large, Fast Instruction Window for Tolerating Cache Misses, ” ISCA 2002. • But, dependents still hold on to the physical registers • Cannot scale the size of the register file indefinitely since it is on the critical path • Possible solution: Deallocate physical registers of dependents – Difficult to re-allocate. See Srinivasan et al, “Continual Flow Pipelines, ” ASPLOS 2004. Can you think of any other solution? 43

FOOD FOR THOUGHT FOR YOU • Many other design choices • Should reservation stations and ROB store data values or should there be a centralized physical register file where all data values are stored? – What are the tradeoffs? • Exactly when does an instruction broadcast its tag? • … 44

MORE FOOD FOR THOUGHT FOR YOU • How can you implement branch prediction in an out-oforder execution machine? – Think about branch history register and PHT updates – Think about recovery from mispredictions • How to do this fast? • How can you combine superscalar execution with out-oforder execution? – These are different concepts – Concurrent renaming of instructions – Concurrent broadcast of tags • How can you combine superscalar + out-of-order + branch prediction? 45

GENERAL ORGANIZATION OF AN OOO PROCESSOR n Smith and Sohi, “The Microarchitecture of Superscalar Processors, ” Proc. IEEE, Dec. 1995. 46

A MODERN OOO DESIGN: INTEL PENTIUM 4 47 Boggs et al. , “The Microarchitecture of the Pentium 4 Processor, ” Intel Technology Journal, 2001.

INTEL PENTIUM 4 SIMPLIFIED Mutlu+, “Runahead Execution, ” HPCA 2003. 48

ALPHA 21264 Kessler, “The Alpha 21264 Microprocessor, ” IEEE Micro, March-April 1999. 49

MIPS R 10000 Yeager, “The MIPS R 10000 Superscalar Microprocessor, ” IEEE Micro, April 1996 50

IBM POWER 4 • Tendler et al. , “POWER 4 system microarchitecture”, IBM J R&D, 2002. 51

IBM POWER 4 • • • 2 cores, out-of-order execution 100 -entry instruction window in each core 8 -wide instruction fetch, issue, execute Large, local+global hybrid branch predictor 1. 5 MB, 8 -way L 2 cache Aggressive stream based prefetching 52

n IBM POWER 5 Kalla et al. , “IBM Power 5 Chip: A Dual-Core Multithreaded Processor, ” IEEE Micro 2004. 53

RECOMMENDED READINGS • Out-of-order execution processor designs • Kessler, “The Alpha 21264 Microprocessor, ” IEEE Micro, March-April 1999. • Boggs et al. , “The Microarchitecture of the Pentium 4 Processor, ” Intel Technology Journal, 2001. • Yeager, “The MIPS R 10000 Superscalar Microprocessor, ” IEEE Micro, April 1996 • Tendler et al. , “POWER 4 system microarchitecture, ” IBM Journal of Research and Development, January 2002. 54

AND MORE READINGS… • Stark et al. , “On Pipelining Dynamic Scheduling Logic, ” MICRO 2000. • Brown et al. , “Select-free Instruction Scheduling Logic, ” MICRO 2001. • Palacharla et al. , “Complexity-effective Superscalar Processors, ” ISCA 1997. 55