Using Prediction to Accelerate Coherence Protocols Shubhendu S

Using Prediction to Accelerate Coherence Protocols Shubhendu S Mukherjee and Mark D Hill University of Wisconsin Madison

The topic once again n Using Prediction to Protocols n n Accelerate Coherence Discuss the concept of using prediction in a coherence protocol See how it can be used to accelerate the protocol

Organization n n Introduction Background n n n Cosmos n n n How and When to act on the predictions Handling Mis-predictions Performance Evaluation n Basic Structure Obtaining Predictions Implementation Issues Integration with a Coherence Protocol n n Directory Protocol Two-level Branch Predictor Benchmarks Results Summary and Conclusions

Introduction n n Large shared memory multi processors suffer from long latencies for misses to remotely cached blocks Proposals to lessen these latencies n n n Multithreading Non-blocking caches Application specific coherence protocols Predict future sharing patterns, overlap execution with coherence work Drawbacks n n n More complex program model Require sophisticated compilers Existing predictors are directed at specific sharing patterns known a priori n Need for a general predictor, hence this paper!

Introduction n If general predictor is not in the army then what is it? A general predictor would sit beside standard directory or cache module, monitor coherence activity and take appropriate actions n See the design of Cosmos coherence message predictor n Evaluate Cosmos on some scientific applications n Alls well that ends well? Summarize and conclude

Background: 6810 strikes back! n Structure of a Directory Protocol n n n n Distributed memory multiprocessor Hardware based cache coherence Directory and memory distributed among processors Physical address gives the location of memory Nodes connected to each other via a scalable interconnect Messages routed from sender to receiver Directory keeps track of sharing states, which are?

Directory Structure Processor & Caches Memory Directory I/O Processor & Caches Memory Directory Interconnection network I/O Processor & Caches Memory Directory I/O

Example: Coherence Protocol Actions Processor 1 & Caches Memory 1 Directory Wr A I/O Processor 2 & Caches Memory 3 5 4 Directory 2 Interconnection network I/O A 1. ? 2. ? 3. ? 4. ? 5. ?

Example: Coherence Protocol Actions Processor 1 & Caches Memory 1 Directory Wr A I/O Processor 2 & Caches Memory 3 5 4 Directory 2 Interconnection network I/O A 1. P 1 Wr request to Dir 1 2. Dir 1 Inval request Dir 2 3. Dir 2 Inval Cache copy P 2 4. Dir 2 Inval response Dir 1 5. Dir 1 Wr response P 1

Example: Coherence Protocol Actions Processor 1 & Caches Memory 1 Directory Wr A I/O Processor 2 & Caches Memory 3 5 4 Directory I/O A 1. P 1 Wr request to Dir 1 2. Dir 1 Inval request Dir 2 3. Dir 2 Inval Cache copy P 2 4. Dir 2 Inval response Dir 1 5. Dir 1 Wr response P 1 2 Interconnection network Point to ponder: Multiple long-latency operations (sequential)

Background: 6810 strikes back! n Branch predictor n n Need: Execute probable instructions without waiting, thus improve performance Two Level n n n Basically a Local predictor Use PC of branch to index into Branch History Table(Local) Use this BHT entry to index into per branch Pattern History Table to obtain a branch prediction

Two Level Predictor Branch PC Use 6 bits of branch PC to index into branch history table 1011011001 Table of 64 entries of 14 -bit histories for a single branch 14 -bit history indexes into next level Table of 16 K entries of 2 -bit saturating counters Pattern History Table

What in Universe is COSMOS? n n n Cosmos is a Coherence Message Predictor Predicts the sender and type of next incoming message for a particular block. Structure : Similar to a two level branch predictor

Structure of Cosmos Message History Register (MHR) Pattern History Tables (Per block address) Message History Table (MHT)

Structure of Cosmos Message History Register (MHR) <sender, type> … Number of tuples per MHR constitutes its depth Message History Table (MHT)

Structure of Cosmos n n n n The first level table is called the Message History Table (MHT) An MHT consists of a series of Message History Registers (MHR) (one per cache block address) An MHR contains a sequence of <sender, type> tuples (depth) The second level table is called the Pattern History Table(PHT) There is one PHT for each MHR PHT is indexed by the entry in MHR Each PHT contains prediction tuples corresponding to MHR entries

An Example: Producer - Consumer repeat … if(producer) private_counter++ shared_counter = private_counter barrier else if(consumer) barrier private_counter = shared_counter else barrier endif … until done

An Example: Producer - Consumer Processor 1 & Caches Memory Directory I/O Producer Processor 2 & Caches Memory Directory Interconnection network I/O Consumer

An Example: Producer - Consumer Producer Cache Memory 1 I/O 2 ? Directory Messages seen by the Producer Cache (from directory) ?

An Example: Producer - Consumer Producer Cache Memory 1 I/O 2 1. Get Wr Response Directory Messages seen by the Producer Cache 2. Invalidate Wr request

An Example: Producer - Consumer Cache Memory 1 I/O 2 ? Directory Messages seen by the Consumer Cache(from directory) ?

An Example: Producer - Consumer Cache Memory 1 I/O 2 1. Get Rd Response Directory Messages seen by the Consumer Cache 2. Invalidate Rd request

An Example: Producer - Consumer ? ? Messages seen by the Directory

An Example: Producer - Consumer 1. Get Wr Request from producer 4. Invalidate Wr Response from producer 2. Invalidate Rd Response from consumer 3. Get Rd Request from consumer Messages seen by the Directory

An Example: Producer - Consumer Sharing Pattern Signature n n Predictable message patterns Producer send Get Wr request to directory receive Get Wr response from directory receive Invalidate Wr request from directory send Invalidate Wr response to directory Consumer send Get Rd request to directory receive Get Rd response from directory receive Invalidate Rd request from directory send Invalidate Rd response to directory

Back to Cosmos • Directory receives get Rd request from the consumer Pattern History Table for shared_counter ? Message History Table Global Address of shared_counter P 1: Producer P 2: Consumer

Back to Cosmos • Directory receives get Rd request from the consumer Pattern History Table for shared_counter Message History Table Global Address of shared_counter P 1: Producer P 2: Consumer

Back to Cosmos n Obtaining Predictions n n Index into MHR table with the address of the cache block Use the MHR entry to index into the corresponding PHT Return the prediction (if one exists) from the PHT. This prediction is of the form < Sender , Message – type >. Updating Cosmos n n Index into MHR table with the address of the cache block Use the MHR entry to index into the corresponding PHT Write new <Sender, Message – type> tuple as prediction for index corresponding to the MHR entry Insert the <Sender, Message – type> tuple into the MHR for the cache block

How Cosmos adapts to complex signatures n Consider one Producer and two Consumers P 1 and P 2 Two get Rd requests arrive out of order. PHT will then be as shown below Index Prediction

How Cosmos adapts to complex signatures MHR with depth greater than 1 Index Prediction

Implementation issues n Storage Issues n n n Possible to merge the first level table with cache block state at cache and the directory? Second level table will need more memory to catch pattern histories for each cache block If number of pattern histories for each cache block is found to be low, per allocate memory for the pattern histories If more pattern histories needed, allocate them from a common pool of dynamically allocated memory Higher prediction accuracies require higher MHR depths : may result in large amounts of memory

Integration with a Coherence protocol n n Predictors sit beside cache and directory module and accelerate coherence activity in two steps: Step 1: Monitor message activity and make a prediction n Step 2: Invoke an action based on the prediction n Key challenges: n n n Knowing how and when to act on the predictions Handling Mis – predictions Performance

How to act on predictions n Some Examples Prediction Location Static / Dynamic Ld/St from Processor Cache Static Pre fetch block in shared or exclusive state Stanford DASH protocol Read – modify - write Directory Static Directory responds with block in exclusive state for read miss for idle block SGI Origin Protocol Read – modify - write Cache Static Cache requests exclusive copy on read miss Dir 1 SW, Dir 1 SW+ Store from different processor Cache Static Replace block and return to directory Dir 1 SW, Dir 1 SW+ Store from different processor Directory Dynamic Invalidate and replace block to directory if exclusive Dynamic self invalidation Block migrates between different processors Directory Dynamic On read miss return block to requesting processor in exclusive state Migratory Protocols Prediction Action Protocol

Detecting and Handling Mis-predictions n Usual problem with predictions n n Mis-predictions may leave processor state / protocol state in an inconsistent state Actions taken after predictions can be classified into three categories n n n Actions that move the protocol between two legal states Actions that move the protocol to a future state, but do not expose this state to the processor Actions that allow both processor and the protocol to move to future states

Handling Mis-Predictions n Actions that move the protocol between two legal states Example : Replacement of a cache block that moves the block from “exclusive” to “invalid” state No explicit recovery in this case P 1 Cache Directory P 2 Cache Time Get Wr request Inval Wr response Get Wr response

Handling Mis-Predictions n Actions that move the protocol to a future state, but do not expose this state to the processor If mis-prediction, simply discard the future state If prediction is correct, commit the future state and expose it to the processor P 1 Cache Directory P 2 Cache Time Get Wr request Inval Wr request Predicts, updates protocol state, generates message Sends message Inval Wr response Get Wr response

Handling Mis-Predictions P 1 Cache Time Directory P 2 Cache Predicts, updates protocol state, generates message Mis-Predict Send correct response

Handling Mis-Predictions n Actions that allow both processor and the protocol to move to future states n n Need greater support for recovery Before speculation, both processor and protocol can checkpoint their states On detecting Mis-predictions , they rollback to the check pointed states On correct prediction, the current protocol and processor states must be committed

Performance n How prediction affects runtime n A simplistic execution model is as follows. Let : p be the prediction accuracy for each message, f be the fraction of delay incurred on messages predicted correctly (e. g. f = 0 means that the time of a message predicted correctly is completely overlapped with other delays), and r be the penalty due to a mis-predicted message (e. g. , r = O. 5 implies a mis-predicted message takes 1. 5 times the delay of a message without prediction).

Performance n How prediction affects runtime p be the prediction accuracy for each message, f be the fraction of delay incurred on messages predicted correctly r be the penalty due to a mis-predicted message If performance is completely determined by the number of messages in the critical path of a parallel program, then speedup due to prediction is: time(w/o prediction) --------------time (with prediction) = 1 --------------p * f + (1 -p) * (1+r)

Performance E. g. : For a prediction accuracy of 80% (p=0. 8), speedup = 56% with a misprediction penalty of 100%(r=1) and a prediction success benefit of 30% (f=0. 3)

Evaluation n Cosmos’ prediction accuracy is evaluated using traces of coherence messages obtained from the Wisconsin Stache protocol running five parallel scientific applications Wisconsin Stache protocol Stache is a software, full-map, and write-invalidate directory protocol that uses part of local memory as a cache for remote data. Benchmarks Five parallel scientific applications: appbt, barnes, dsmc, moldyn, unstructured

Benchmarks n Appbt is a parallel three-dimensional computational fluid dynamics application. n Barnes simulates the interaction of a system of bodies in three dimensions using the Barnes-Hut hierarchical N-body method. n Dsmc studies the properties of a gas by simulating the movement and collision of a large number of particles in a three-dimensional domain with discrete simulation Monte Carlo method.

Benchmarks n Moldyn is a molecular dynamics application. n Unstructured is a computational fluid dynamics application that uses an unstructured mesh to model a physical structure, such as an airplane wing or body.

Results appbt Depth of MHR 1 2 3 4 barnes dsmc moldyn unstructured C D O C D O 91 90 89 89 77 79 80 80 84 85 85 85 C: cache prediction rate D: Directory prediction rate O: Overall prediction rate 80 81 79 78 42 56 57 56 62 69 69 68 94 95 94 94 73 77 92 92 84 86 93 93 92 91 90 90 79 80 79 77 86 86 85 84 85 90 90 96 65 86 88 88 74 88 89 92

Results: Observations n Overall prediction accuracy : 62 ~ 86% Higher accuracy for cache compared to directory: Why ? Prediction accuracy increases with an increase in MHR depth However, not much increase beyond MHR depth of 3 n Appbt: n n n High prediction accuracy Producer-consumer sharing pattern Producer reads, writes and consumer reads n Barnes : Lower accuracy than other applications Nodes of octree are assigned different shared memory addresses in different iterations

Results: Observations n Dsmc: Highest accuracy among all applications Producer-consumer sharing patterns Producer writes and consumer reads Why higher than Appbt? n Moldyn: High accuracy Migratory and producer-consumer sharing patterns n Unstructured: Different dominant signatures for same data structures in different phases of the application Migratory and producer-consumer sharing patterns

Effects of noise-filters n n Remember them? Cosmos noise filter: Saturating counter : 0 to MAXCOUNT, here till 2 MHR depth >2, filters do not help much – Why? Predictors with MHR>1 can adapt to noise, greater accuracy for repeating noise appbt Depth of MHR 1 2 barnes dsmc moldyn unstructured 0 1 2 0 1 2 84 85 85 86 62 69 0 , 1, 2: MAXCOUNT 66 71 84 86 86 88 86 86 86 74 88 78 89

Summary and Conclusions n Comparison with directed optimizations n Worse: Less cost effective as more hardware required n Better: n n Including the composition of predictors of several directed optimizations in a single protocol will be more complex than a single Cosmos Can discover application-specific sharing patterns not known a priori

Summary and Conclusions n n n We explored using Prediction to Accelerate coherence protocol Protocol executes faster if future actions can be predicted and executed speculatively. We came across Cosmos n n We evaluated Cosmos using scientific applications n n High prediction accuracy because of predictable coherence message patterns Cosmos is more general than directed optimizations n n MHT, MHR, PHT Two-level predictor Use <sender, message-type> tuple Can be costly because of large resource usage Can be easily integrated with a protocol Can discover and track application specific patterns not known a priori Finally, more work is needed to determine if the high prediction rates can be used to significantly reduce execution time with a coherence protocol.

Questions ?