Simultaneous Multithreading SMT An evolutionary processor architecture originally

Simultaneous Multithreading (SMT) • An evolutionary processor architecture originally introduced in 1996 by Dean Tullsen at the University of Washington that aims at reducing resource waste in wide issue processors. • SMT has the potential of greatly enhancing processor computational capabilities by: – Exploiting thread-level parallelism (TLP), simultaneously executing instructions from different threads during the same cycle. – Providing multiple hardware contexts, hardware thread scheduling and context switching capability. EECC 722 - Shaaban #1 Lec # 2 Fall 2001 9 -10 -2001

SMT Issues • SMT CPU performance gain potential. • Modifications to Superscalar CPU architecture necessary to support SMT. • SMT performance evaluation vs. Fine-grain multithreading Superscalar, Chip Multiprocessors. • Hardware techniques to improve SMT performance: – Optimal level one cache configuration for SMT. – SMT thread instruction fetch, issue policies. – Instruction recycling (reuse) of decoded instructions. • Software techniques: – Compiler optimizations for SMT. – Software-directed register deallocation. – Operating system behavior and optimization. • SMT support for fine-grain synchronization. • SMT as a viable architecture for network processors. EECC 722 - Shaaban #2 Lec # 2 Fall 2001 9 -10 -2001

Microprocessor Architecture Trends EECC 722 - Shaaban #3 Lec # 2 Fall 2001 9 -10 -2001

Performance Increase of Workstation-Class Microprocessors 1987 -1997 Integer SPEC 92 Performance EECC 722 - Shaaban #4 Lec # 2 Fall 2001 9 -10 -2001

Microprocessor Logic Density Moore’s Law Alpha 21264: 15 million Pentium Pro: 5. 5 million Power. PC 620: 6. 9 million Alpha 21164: 9. 3 million Sparc Ultra: 5. 2 million Moore’s Law: 2 X transistors/Chip Every 1. 5 years EECC 722 - Shaaban #5 Lec # 2 Fall 2001 9 -10 -2001

Increase of Capacity of VLSI Dynamic RAM Chips year size(Megabit) 1980 1983 1986 1989 1992 1996 1999 2000 0. 0625 0. 25 1 4 16 64 256 1024 1. 55 X/yr, or doubling every 1. 6 years EECC 722 - Shaaban #6 Lec # 2 Fall 2001 9 -10 -2001

CPU Architecture Evolution: Single Threaded Pipeline • Traditional 5 -stage pipeline. • Increases Throughput: Ideal CPI = 1 EECC 722 - Shaaban #7 Lec # 2 Fall 2001 9 -10 -2001

CPU Architecture Evolution: Superscalar Architectures • Fetch, decode, execute, etc. more than one instruction per cycle (CPI < 1). • Limited by instruction-level parallelism (ILP). EECC 722 - Shaaban #8 Lec # 2 Fall 2001 9 -10 -2001

Superscalar Architectures: Issue Slot Waste Classification • Empty or wasted issue slots can be defined as either vertical waste or horizontal waste: – Vertical waste is introduced when the processor issues no instructions in a cycle. – Horizontal waste occurs when not all issue slots can be filled in a cycle. EECC 722 - Shaaban #9 Lec # 2 Fall 2001 9 -10 -2001

Sources of Unused Issue Cycles in an 8 -issue Superscalar Processor busy represents the utilized issue slots; all others represent wasted issue slots. 61% of the wasted cycles are vertical waste, the remainder are horizontal waste. Workload: SPEC 92 benchmark suite. Source: Simultaneous Multithreading: Maximizing On-Chip Parallelism Dean Tullsen et al. , Proceedings of the 22 rd Annual International Symposium on Computer Architecture, June 1995, pages 392 -403. EECC 722 - Shaaban #10 Lec # 2 Fall 2001 9 -10 -2001

Superscalar Architectures: All possible causes of wasted issue slots, and latency-hiding or latency reducing techniques that can reduce the number of cycles wasted by each cause. Source: Simultaneous Multithreading: Maximizing On-Chip Parallelism Dean Tullsen et al. , Proceedings of the 22 rd Annual International Symposium on Computer Architecture, June 1995, pages 392 -403. EECC 722 - Shaaban #11 Lec # 2 Fall 2001 9 -10 -2001

Advanced CPU Architectures: Fine-grain or Traditional Multithreaded Processors • Multiple HW contexts (PC, SP, and registers). • One context gets CPU for x cycles at a time. • Limited by thread-level parallelism (TLP): – Can reduce some of the vertical issue slot waste. – No reduction in horizontal issue slot waste. • Example Architectures: HEP, Tera. EECC 722 - Shaaban #12 Lec # 2 Fall 2001 9 -10 -2001

Advanced CPU Architectures: VLIW: Intel/HP Explicitly Parallel Instruction Computing (EPIC) • Strengths: – Allows for a high level of instruction parallelism (ILP). – Takes a lot of the dependency analysis out of HW and places focus on smart compilers. • Weakness: – Limited by instruction-level parallelism (ILP) in a single thread. – Keeping Functional Units (FUs) busy (control hazards). – Static FUs Scheduling limits performance gains. EECC 722 - Shaaban #13 Lec # 2 Fall 2001 9 -10 -2001

Advanced CPU Architectures: Single Chip Multiprocessor • Strengths: – Create a single processor block and duplicate. – Takes a lot of the dependency analysis out of HW and places focus on smart compilers. • Weakness: – Performance limited by individual thread performance (ILP). EECC 722 - Shaaban #14 Lec # 2 Fall 2001 9 -10 -2001

Advanced CPU Architectures: Single Chip Multiprocessor EECC 722 - Shaaban #15 Lec # 2 Fall 2001 9 -10 -2001

SMT: Simultaneous Multithreading • Multiple Hardware Contexts running at the same time (HW context: registers, PC, and SP). • Avoids both horizontal and vertical waste by having multiple threads keeping functional units busy during every cycle. • Builds on top of current time-proven advancements in CPU design: superscalar, dynamic scheduling, hardware speculation, dynamic HW branch prediction. • Enabling Technology: VLSI logic density in the order of hundreds of millions of transistors/Chip. EECC 722 - Shaaban #16 Lec # 2 Fall 2001 9 -10 -2001

SMT • With multiple threads running penalties from long-latency operations, cache misses, and branch mispredictions will be hidden: – Reduction of both horizontal and vertical waste and thus improved Instructions Issued Per Cycle (IPC) rate. • Pipelines are separated until issue stage. • Functional units are shared among all contexts during every cycle: – More complicated writeback stage. • More threads issuing to functional units results in higher resource utilization. EECC 722 - Shaaban #17 Lec # 2 Fall 2001 9 -10 -2001

SMT: Simultaneous Multithreading EECC 722 - Shaaban #18 Lec # 2 Fall 2001 9 -10 -2001

Time (processor cycles) The Power Of SMT 1 1 1 1 2 2 2 3 3 3 4 2 2 4 4 5 1 1 1 2 2 3 1 2 4 1 2 5 4 1 1 1 5 5 1 1 1 2 2 2 1 3 1 4 4 4 Superscalar Traditional Multithreaded Rows of squares represent instruction issue slots Box with number x: instruction issued from thread x Empty box: slot is wasted 5 1 Simultaneous Multithreading EECC 722 - Shaaban #19 Lec # 2 Fall 2001 9 -10 -2001

SMT Performance Example Inst A B C D E F G H I J K • • • Code LUI FMUL ADD MUL LW ADD NOT FADD XOR SUBI SW R 5, 100 F 1, F 2, F 3 R 4, 8 R 3, R 4, R 5 R 6, R 4 R 1, R 2, R 3 R 7, R 7 F 4, F 1, F 2 R 8, R 1, R 7 R 2, R 1, 4 ADDR, R 2 Description R 5 = 100 F 1 = F 2 x F 3 R 4 = R 4 + 8 R 3 = R 4 x R 5 R 6 = (R 4) R 1 = R 2 + R 3 R 7 = !R 7 F 4=F 1 + F 2 R 8 = R 1 XOR R 7 R 2 = R 1 – 4 (ADDR) = R 2 Functional unit Int ALU FP ALU Int mul/div Memory port Int ALU FP ALU Int ALU Memory port 4 integer ALUs (1 cycle latency) 1 integer multiplier/divider (3 cycle latency) 3 memory ports (2 cycle latency, assume cache hit) 2 FP ALUs (5 cycle latency) Assume all functional units are fully-pipelined EECC 722 - Shaaban #20 Lec # 2 Fall 2001 9 -10 -2001

SMT Performance Example (continued) • • 2 additional cycles to complete program 2 Throughput: – Superscalar: 11 inst/7 cycles = 1. 57 IPC – SMT: 22 inst/9 cycles = 2. 44 IPC EECC 722 - Shaaban #21 Lec # 2 Fall 2001 9 -10 -2001

Changes to Superscalar CPUs Necessary to support SMT • Multiple program counters and some mechanism by which one fetch unit selects one each cycle (thread instruction fetch policy). • A separate return stack for each thread for predicting subroutine return destinations. • Per-thread instruction retirement, instruction queue flush, and trap mechanisms. • A thread id with each branch target buffer entry to avoid predicting phantom branches. • A larger register file, to support logical registers for all threads plus additional registers for register renaming. (may require additional pipeline stages). • A higher available main memory fetch bandwidth may be required. • Improved cache to offset the cache performance degradation due to cache sharing among the threads and the resulting reduced locality. – e. g Private per-thread vs. shared L 1 cache. EECC 722 - Shaaban #22 Lec # 2 Fall 2001 9 -10 -2001

A Base SMT hardware Architecture. Source: Exploiting Choice: Instruction Fetch and Issue on an Implementable Simultaneous Multithreading Processor, Dean Tullsen et al. Proceedings of the 23 rd Annual International Symposium on Computer Architecture, May 1996, pages 191 -202. EECC 722 - Shaaban #23 Lec # 2 Fall 2001 9 -10 -2001

Example SMT Vs. Superscalar Pipeline • The pipeline of (a) a conventional superscalar processor and (b) that pipeline modified for an SMT processor, along with some implications of those pipelines. Source: Exploiting Choice: Instruction Fetch and Issue on an Implementable Simultaneous Multithreading Processor, Dean Tullsen et al. Proceedings of the 23 rd Annual International Symposium on Computer Architecture, May 1996, pages 191 -202. EECC 722 - Shaaban #24 Lec # 2 Fall 2001 9 -10 -2001

SMT Performance Comparison • Instruction throughput from simulations by Eggers et al. at The University of Washington, using both multiprogramming and parallel workloads: Multiprogramming workload Superscalar Threads 1 2 4 8 2. 7 - Traditional Multithreading 2. 6 3. 3 3. 6 2. 8 SMT 3. 1 3. 5 5. 7 6. 2 Parallel Workload Superscalar Threads 1 2 4 8 3. 3 - MP 2 MP 4 2. 4 4. 3 - 1. 5 2. 6 4. 2 - Traditional Multithreading 3. 3 4. 1 4. 2 3. 5 SMT 3. 3 4. 7 5. 6 6. 1 EECC 722 - Shaaban #25 Lec # 2 Fall 2001 9 -10 -2001

Simultaneous Vs. Fine-Grain Multithreading Performance Instruction throughput as a function of the number of threads. (a)-(c) show the throughput by thread priority for particular models, and (d) shows the total throughput for all threads for each of the six machine models. The lowest segment of each bar is the contribution of the highest priority thread to the total throughput. Source: Simultaneous Multithreading: Maximizing On-Chip Parallelism Dean Tullsen et al. , Proceedings of the 22 rd Annual International Symposium on Computer Architecture, June 1995, pages 392 -403. EECC 722 - Shaaban #26 Lec # 2 Fall 2001 9 -10 -2001

Simultaneous Multithreading Vs. Single-Chip Multiprocessing • Results for the multiprocessor MP vs. simultaneous multithreading SM comparisons. The multiprocessor always has one functional unit of each type per processor. In most cases the SM processor has the same total number of each FU type as the MP. Source: Simultaneous Multithreading: Maximizing On-Chip Parallelism Dean Tullsen et al. , Proceedings of the 22 rd Annual International Symposium on Computer Architecture, June 1995, pages 392 -403. EECC 722 - Shaaban #27 Lec # 2 Fall 2001 9 -10 -2001

Impact of Level 1 Cache Sharing on SMT Performance • Results for the simulated cache configurations, shown relative to the throughput (instructions per cycle) of the 64 s. 64 p • The caches are specified as: [total I cache size in KB][private or shared]. [D cache size][private or shared] For instance, 64 p. 64 s has eight private 8 KB I caches and a shared 64 KB data Source: Simultaneous Multithreading: Maximizing On-Chip Parallelism Dean Tullsen et al. , Proceedings of the 22 rd Annual International Symposium on Computer Architecture, June 1995, pages 392 -403. EECC 722 - Shaaban #28 Lec # 2 Fall 2001 9 -10 -2001

SMT Thread Instruction Fetch Scheduling Policies • Round Robin: – Instruction from Thread 1, then Thread 2, then Thread 3, etc. (eg RR 1. 8 : each cycle one thread fetches up to eight instructions RR 2. 4 each cycle two threads fetch up to four instructions each) • BR-Count: – Give highest priority to those threads that are least likely to be on a wrong path by by counting branch instructions that are in the decode stage, the rename stage, and the instruction queues, favoring those with the fewest unresolved branches. • MISS-Count: – Give priority to those threads that have the fewest outstanding Data cache misses. • I-Count: – Highest priority assigned to thread with the lowest number of instructions in static portion of pipeline (decode, rename, and the instruction queues). • IQPOSN: – Give lowest priority to those threads with instructions closest to the head of either the integer or floating point instruction queues (the oldest instruction is at the head of the queue). EECC 722 - Shaaban #29 Lec # 2 Fall 2001 9 -10 -2001

Instruction throughput & Thread Fetch Policy EECC 722 - Shaaban #30 Lec # 2 Fall 2001 9 -10 -2001

Possible SMT Instruction Issue Policies • OLDEST FIRST: Issue the oldest instructions (those deepest into the instruction queue). • OPT LAST and SPEC LAST: Issue optimistic and speculative instructions after all others have been issued. • BRANCH FIRST: Issue branches as early as possible in order to identify mispredicted branches quickly. Source: Exploiting Choice: Instruction Fetch and Issue on an Implementable Simultaneous Multithreading Processor, Dean Tullsen et al. Proceedings of the 23 rd Annual International Symposium on Computer Architecture, May 1996, pages 191 -202. EECC 722 - Shaaban #31 Lec # 2 Fall 2001 9 -10 -2001

Simulator (sim-SMT) @ RIT CE • • Execution-driven, performance simulator. Derived from Simple Scalar tool set. Simulates cache, branch prediction, five pipeline stages Flexible: – Configuration File controls cache size, buffer sizes, number of functional units. • Cross compiler used to generate Simple Scalar assembly language. • Binary utilities, compiler, and assembler available. • Standard C library (libc) has been ported. EECC 722 - Shaaban #32 Lec # 2 Fall 2001 9 -10 -2001

Simulator Memory Address Space EECC 722 - Shaaban #33 Lec # 2 Fall 2001 9 -10 -2001

Alternate Functional Unit Configurations • New functional unit configurations attempted (by adding one of each type of FU): – +1 integer multiplier/divider • +2. 8% IPC, issue rate • -74% times with no FU available • Simulator very flexible (only one line in configuration file required change) EECC 722 - Shaaban #34 Lec # 2 Fall 2001 9 -10 -2001

Sim-SMT Simulator Limitations • Does not keep precise exceptions. • System Call’s instructions not tracked. • Limited memory space: – Four test programs’ memory spaces running on one simulator memory space – Easy to run out of stack space EECC 722 - Shaaban #35 Lec # 2 Fall 2001 9 -10 -2001

Simulation Runs & Results • Test Programs used: – – Newton interpolation. Matrix Solver using LU decomposition. Integer Test Program. FP Test Program. • Simulations of a single program – 1, 2, and 4 threads. • System simulations involve a combination of all programs simultaneously – Several different combinations were run • From simulation results: – Performance increase: • Biggest increase occurs when changing from one to two threads. – Higher issue rate, functional unit utilization. EECC 722 - Shaaban #36 Lec # 2 Fall 2001 9 -10 -2001

Simulation Results: Performance (IPC) EECC 722 - Shaaban #37 Lec # 2 Fall 2001 9 -10 -2001

Simulation Results: Simulation Time EECC 722 - Shaaban #38 Lec # 2 Fall 2001 9 -10 -2001

Simulation Results: Instruction Issue Rate EECC 722 - Shaaban #39 Lec # 2 Fall 2001 9 -10 -2001

Simulation Results: Performance Vs. Issue BW EECC 722 - Shaaban #40 Lec # 2 Fall 2001 9 -10 -2001

Simulation Results: Functional Unit Utilization EECC 722 - Shaaban #41 Lec # 2 Fall 2001 9 -10 -2001

Simulation Results: No Functional Unit Available EECC 722 - Shaaban #42 Lec # 2 Fall 2001 9 -10 -2001

Simulation Results: Horizontal Waste Rate EECC 722 - Shaaban #43 Lec # 2 Fall 2001 9 -10 -2001

Simulation Results: Vertical Waste Rate EECC 722 - Shaaban #44 Lec # 2 Fall 2001 9 -10 -2001

SMT: Simultaneous Multithreading • Strengths: – Overcomes the limitations imposed by low single thread instruction-level parallelism. – Multiple threads running will hide individual control hazards (branch mispredictions). • Weaknesses: – Additional stress placed on memory hierarchy Control unit complexity. – Sizing of resources (cache, branch prediction, etc. ) – Accessing registers (32 integer + 32 FP for each HW context): • Some designs devote two clock cycles for both register reads and register writes. EECC 722 - Shaaban #45 Lec # 2 Fall 2001 9 -10 -2001

SMT: Simultaneous Multithreading Kernel Code • Many, if not all, benchmarks are based upon a limited interaction with kernel code. • How can the kernel overhead be minimized (contextswitching, process management, etc. )? – CHAOS (Context Hardware Accelerated Operating System). • Introduce a lightweight dedicated kernel context to handle process management: – When there are 4 contexts, there is a good chance that one of them will continue to run, why take an (expensive) chance in swapping it out when it will be brought right back in by the swapper (process management). EECC 722 - Shaaban #46 Lec # 2 Fall 2001 9 -10 -2001

SMT & Technology • SMT architecture has not been implemented in any existing commercial microprocessor yet (First 4 -thread SMT CPU: Alpha EV 8 ~2001). • Current technology has the potential for 4 -8 simultaneous threads: – Based on transistor count and design complexity. EECC 722 - Shaaban #47 Lec # 2 Fall 2001 9 -10 -2001

RIT-CE SMT Project Goals • Investigate performance gains from exploiting Thread-Level Parallelism (TLP) in addition to current Instruction-Level Parallelism (ILP) in processor design. • Design and simulate an architecture incorporating Simultaneous Multithreading (SMT). • Study operating system and compiler modifications needed to support SMT processor architectures. • Define a standard interface for efficient SMT-processor/OS kernel interaction. • Modify an existing OS kernel (Linux? ) to take advantage of hardware multithreading capabilities. • Long term: VLSI implementation of an SMT prototype. EECC 722 - Shaaban #48 Lec # 2 Fall 2001 9 -10 -2001

Current Project Status • Architecture/OS interface definition. • Study of design alternatives and impact on performance. • SMT Simulator Development: – System call development, kernel support, and compiler/assembler changes. • Development of code (programs and OS kernel) is key to getting results. EECC 722 - Shaaban #49 Lec # 2 Fall 2001 9 -10 -2001

Short-Term Project Chart EECC 722 - Shaaban #50 Lec # 2 Fall 2001 9 -10 -2001

Current/Future Project Goals • SMT simulator completion refinement, and further testing. • Development of an SMT-capable OS kernel. • Extensive performance studies with various workloads using the simulator/OS/compiler: – Suitability for fine-grained parallel applications? – Effect on multimedia applications? • Architectural changes based on benchmarks. • Cache impact on SMT performance investigation. • Investigation of an in-order SMT processor (C or VHDL model) • MOSIS Tiny Chip (partial/full) implementation. • Investigate the suitability of SMT processors as building blocks for MPPs. EECC 722 - Shaaban #51 Lec # 2 Fall 2001 9 -10 -2001