Hi DISC A Decoupled Architecture for Applications in

Hi. DISC: A Decoupled Architecture for Applications in Data Intensive Computing PIs: Alvin M. Despain and Jean-Luc Gaudiot DARPA DIS PI Meeting Santa Fe, NM March 26 -27, 2002

Hi. DISC: Hierarchical Decoupled Instruction Set Computer New Ideas Sensor Inputs Application (FLIR SAR VIDEO ATR /SLD Scientific ) • A dedicated processor for each level of the memory hierarchy Decoupling Compiler • Explicit management of each level of the memory hierarchy using instructions generated by the compiler • Hide memory latency by converting data access predictability to data access locality Dynamic Database Processor Registers Cache • Exploit instruction-level parallelism without extensive scheduling hardware • Zero overhead prefetches for maximal computation throughput Hi. DISC Processor Memory Situational Awareness Impact Schedule • 2 x speedup for scientific benchmarks with large data sets over an inorder superscalar processor • 7. 4 x speedup for matrix multiply over an in-order issue superscalar processor • Defined benchmarks • Completed simulator • Performed instruction-level • Continue simulations of more benchmarks (SAR) • Define Hi. DISC architecture simulations on hand-compiled • Benchmark result benchmarks • Update Simulator • Developed and tested a complete decoupling compiler • Generated performance statistics and evaluated design • 2. 6 x speedup for matrix decomposition/substitution over an in-order issue superscalar processor • Reduced memory latency for systems that have high memory bandwidths (e. g. PIMs, RAMBUS) • Allows the compiler to solve indexing functions for irregular applications • Reduced system cost for high-throughput scientific codes Start April 2001 March 2002

Accomplishments n n Design of the Hi. DISC model Compiler development (operational) Simulator design (operational) Performance Evaluation q q n Results q n n Three DIS benchmarks (Multidimensional Fourier Transform, Method of Moments, Data Management) Five stressmarks (Pointer, Transitive Closure, Neighborhood, Matrix, Field) Mostly higher performance (some lower) Range of applications Hi. DISC of the future

Outline n n Original Technical Objective Hi. DISC Architecture Review Benchmark Results Conclusion and Future Works

Hi. DISC: Hierarchical Decoupled Instruction Set Computer Sensor Inputs Application (FLIR SAR VIDEO ATR /SLD Scientific ) Decoupling Compiler Dynamic Database Processor Registers Cache Hi. DISC Processor Memory Situational Awareness n n Technological Trend: Memory latency is getting longer relative to microprocessor speed (40% per year) Problem: Some SPEC benchmarks spend more than half of their time stalling [Lebeck and Wood 1994] Domain: benchmarks with large data sets: symbolic, signal processing and scientific programs Present Solutions: Larger Caches, Prefetching (software & hardware), Simultaneous Multithreading

Present Solutions Solution Limitations Larger Caches — — Hardware Prefetching — — Software Prefetching — — — Multithreading — Slow Works well only if working set fits cache and there is temporal locality. Cannot be tailored for each application Behavior based on past and present execution-time behavior Ensure overheads of prefetching do not outweigh the benefits > conservative prefetching Adaptive software prefetching is required to change prefetch distance during run-time Hard to insert prefetches for irregular access patterns Solves the throughput problem, not the memory latency problem

The Hi. DISC Approach Observation: • Software prefetching impacts compute performance • PIMs and RAMBUS offer a high-bandwidth memory system - useful for speculative prefetching Approach: • Add a processor to manage prefetching hide overhead • Compiler explicitly manages the memory hierarchy • Prefetch distance adapts to the program runtime behavior

What is Hi. DISC? n A dedicated processor for each level of the memory hierarchy n Explicitly manage each level of the memory hierarchy using instructions generated by the compiler n Hide memory latency by converting data access predictability to data access locality (Just in Time Fetch) n Exploit instruction-level parallelism without extensive scheduling hardware n Zero overhead prefetches for maximal computation throughput Computation Processor (CP) 2 -issue Registers Load Data Queue Slip Control Queue Access Processor (AP) 3 -issue Store Address Queue Store Data Queue L 1 Cache Mgmt. Processor (CMP) 3 -issue L 2 Cache and Higher Level Hi. DISC

Slip Control Queue n The Slip Control Queue (SCQ) adapts dynamically if (prefetch_buffer_full ()) Don’t change size of SCQ; else if ((2*late_prefetches) > useful_prefetches) Increase size of SCQ; else Decrease size of SCQ; q q Late prefetches = prefetched data arrived after load had been issued Useful prefetches = prefetched data arrived before load had been issued

Decoupling Programs for Hi. DISC (Discrete Convolution - Inner Loop) while (not EOD) y = y + (x * h); send y to SDQ Computation Processor Code for (j = 0; j < i; ++j) y[i]=y[i]+(x[j]*h[i-j-1]); Inner Loop Convolution for (j = 0; j < i; ++j) { load (x[j]); load (h[i-j-1]); GET_SCQ; } send (EOD token) send address of y[i] to SAQ Access Processor Code SAQ: Store Address Queue SDQ: Store Data Queue SCQ: Slip Control Queue EOD: End of Data for (j = 0; j < i; ++j) { prefetch (x[j]); prefetch (h[i-j-1]; PUT_SCQ; } Cache Management Code

General View of the Compiler Source Program Gcc Binary Code Disassembler Stream Separator Computation Code Access Code Hi. DISC Compilation Overview Cache Mgmt. Code

Hi. DISC Stream Separator Sequential Source Data Dependency Graph Defining Load/Store Instructions Instruction Chasing for Backward Slice Computation Stream Access Stream Insert Communication Instructions Computation Code Access Code Insert prefetch Instructions Cache Management Code

Stream Separation: Backward Load/Store Chasing LLL 1 n Load/store instructions and “backward slice” are included as Access stream n Other remaining instructions are sent to the Computation stream n Communications between the two streams are also defined at this point

Stream Separation: Creating an Access Stream n n n Communication needs to take place via the various hardware queues. Insert Communication instructions CMP instructions are copied from access stream instructions, except that load instructions are replaced by prefetch instructions

DIS Benchmarks n Application oriented benchmarks q Main characteristic: Large data sets – non contiguous memory access and no temporal locality

DIS Benchmarks Description Method of Moments (Mo. M) Computing the electromagnetic scattering from complex objects Contains computational complexity and requests high memory speed Multidimension Wide range of application usage al Fourier Image processing, synthesis, convolution deconvolution and Transform digital signal filtering (FFT) Data Management (DM) Image Understanding SAR Ray Tracing Done Traditional DBMS processing. Focusing on index algorithms and ad hoc query processing. Done Sampling contains algorithms that perform spatial filtering and data reduction. Morphological filter component, a region of interest (ROI) selection component, and a feature extraction component Compiled (Input file problem) Cost-effective alternative to real data collections SAR Ray-Tracing Benchmark utilizes an image-domain approach to SAR image simulation Done

Atlantic Aerospace Stressmark Suite n Smaller and more specific procedures q q n Seven individual data intensive benchmarks Directly illustrate particular elements of the DIS problem Fewer computation operations than DIS benchmarks q Focusing on irregular memory accesses

Stressmark Suite Name of Stressmark Problem Memory Access Pointer following Small blocks at unpredictable locations. Can be parallelized Done Update Pointer following with memory update Small blocks at unpredictable location Done Matrix Conjugate gradient simultaneous equation solver Dependent on matrix representation Likely to be irregular or mixed, with mixed levels of reuse Done Neighborho od Calculate image texture measures by finding sum and difference histograms Regular access to pairs of words at arbitrary distances Done Field Collect statistics on large field of words Regular, with little re-use Done Corner-Turn Matrix transposition Block movement between processing nodes with N/A practically nil computation Transitive Closure Find all-pairs-shortest-path solution Dependent on matrix representation, but requires Done for a directed graph reads and writes to different matrices concurrently * DIS Stressmark Suite Version 1. 0, Atlantic Aerospace Division

Pointer Stressmarks n n n Basic idea: repeatedly follow pointers to randomized locations in memory Memory access pattern is unpredictable (input data dependent) Randomized memory access pattern: q n Not sufficient temporal and spatial locality for conventional cache architectures Hi. DISC architecture should provide lower average memory access latency q The pointer chasing in AP can run ahead without blocking by the CP

Simulation Parameters (simplescalar 3. 0, 2000) Branch predict mode Bimodal Branch table size 2048 Issue width 4 Window size for superscalar RUU =16 LSQ = 8 Slip distance for AP/EP 50 Data L 1 cache configuration 128 sets, 32 block, 4 -way set associative , LRU Data L 1 cache latency 1 Unified L 2 cache configuration 1024 sets, 64 block, 4 - way set associative, LRU Unified L 2 cache latency 6 Integer functional units ALU( x 4), MUL/DIV Floating point functional units ALU( x 4), MUL/DIV Number of memory ports 2

Superscalar and Hi. DISC n n n Ability to vary the memory latency Superscalar supports OOO issue with 16 RUU (Register Update Unit) and 8 LSQ (Load Store Queue) Each of AP and CP issues instructions in order

DIS Benchmark/Stressmark Results DIS Benchmarks FFT Mo. M ↑ DM IU RAY ↑ N/A ↑ ↑ ↑: better with Hi. DISC ↓: better with Superscalar Stressmarks Pointer Tran. Cl. Field Matrix Neighbor Update Corner ↑ ↑ ↓ ↓ N/A

DIS Benchmark Results

DIS Benchmark Performance n DIS benchmarks perform extremely well in general with our decoupled processing for the following reasons: q q Many long latency floating-point operations Robust with longer memory latency (eg, Method of Moment is a more stream-like process) In FFT, the Hi. DISC architecture also suffers from longer memory latencies (Due to the data dependencies between two streams) DM is not affected by the longer memory latency in either case.

Stressmark Results

Stressmark Results – Good Performance n n n Pointer chasing can be executed far ahead by using decoupled access stream It does not requires the computation results from the CP Transitive closure also produces good results q Not much in the AP depend on the results of CP n AP can run ahead

Some Weak Performance

Performance Bottleneck n n Too much synchronization causes loss of decoupling Unbalanced code size between two streams q n Stressmark suite is highly “access-heavy” Application domain for decoupled architectures: q Balanced ratio between computation and memory access

Synthesis n Synchronization degrades performance q q q n n When the dependency of AP on CP increases, the slip distance of AP cannot be increased More stream like applications will benefit from using Hi. DISC Multithreading support is needed Applications should contain enough computation (1 to 1 ratio) to hide the memory access latency CMP should be simple q q It executes redundant operations if the data is already in cache Cache pollution can occur

Flexi-DISC n n Fundamental characteristics: q inherently highly dynamic at execution time. Dynamic reconfigurable central computational kernel (CK) Multiple levels of caching and processing around CK q adjustable prefetching Multiple processors on a chip which will provide for a flexible adaptation from multiple to single processors and horizontal sharing of the existing resources.

Flexi-DISC n n Partitioning of Computation Kernel q It can be allocated to the different portions of the application or different applications CK requires separation of the next ring to feed it with data The variety of target applications makes the memory accesses unpredictable Identical processing units for outer rings q Highly efficient dynamic partitioning of the resources and their run-time allocation can be achieved