Multiprocessors Flynn Categories Large vs Small Scale Cache

Multiprocessors— Flynn Categories, Large vs. Small Scale, Cache Coherency Professor Alvin R. Lebeck Computer Science 220 Fall 2001

What is Parallel Computer Architecture? • A Parallel Computer is a collection of processing elements that cooperate to solve large problems fast – – – – how large a collection? how powerful are the elements? how does it scale up? how do they cooperate and communicate? how is data transmitted between processors? what are the primitive abstractions? how does it all translate to performance? © Alvin R. Lebeck 1999 2

Parallel Computation: Why and Why Not? • Pros – – Performance Cost-effectiveness (commodity parts) Smooth upgrade path Fault Tolerance • Cons – Difficult to parallelize applications – Requires automatic parallelization or parallel program development – Software! AAHHHH! © Alvin R. Lebeck 1999 3

Flynn Categories • SISD (Single Instruction Single Data) – Uniprocessors • MISD (Multiple Instruction Single Data) – ? ? ? • SIMD (Single Instruction Multiple Data) – Examples: Illiac-IV, CM-2, Intel MMX » Simple programming model » Low overhead » Flexibility » All custom processors • MIMD (Multiple Instruction Multiple Data) – Examples: Intel 4 -way SMP, SUN ES 3000, SGI Origin, Cray T 3 D » Flexible » Use off-the-shelf microprocessors © Alvin R. Lebeck 1999 4

Communication Models • Shared Memory – Processors communicate with shared address space – Easy on small-scale machines – Advantages: » Model of choice for uniprocessors, small-scale MPs » Ease of programming » Lower latency » Easier to use hardware controlled caching • Message passing – Processors have private memories, communicate via messages – Advantages: » Less hardware, easier to design » Focuses attention on costly non-local operations • Can support either model on either HW base © Alvin R. Lebeck 1999 5

Simple Problem for i = 1 to N A[i] = (A[i] + B[i]) * C[i] sum = sum + A[i] • How do you make this loop parallel to run on many processors? © Alvin R. Lebeck 1999 6

Simple Problem • Split the loops // Independent iterations // Run on up to N processors for i = 1 to N A[i] = (A[i] + B[i]) * C[i] // One last loop to run on one processor for i = 1 to N sum = sum + A[i] © Alvin R. Lebeck 1999 7

Small Scale Shared Memory Multiprocessors Cache(s) and TLB P P P P $ $ $ $ Main Memory 0 • • • N-1 Small number of processors connected to one shared memory Memory is equidistant from all processors (UMA) Kernel can run on any processor (symmetric MP) Intel dual/quad Pentium, IBM, SUN, Compaq, almost everyone Some are moving on-chip (e. g. , IBM) © Alvin R. Lebeck 1999 8

Large Scale Shared Memory Multiprocessors Shared • 100 s to 1000 s of nodes (processors) with single shared physical address space • Use General Purpose Interconnection Network – Still have cache coherence protocol – Use messages instead of bus transactions – No hardware broadcast • Communication Assist • Cray T 3 D, T 3 E, Compaq EV 7, SUN ES 3000 © Alvin R. Lebeck 1999 P P Mem P $ $ Mem P Cntrl/NI $ $ Mem Cntrl/NI Interconnect 9

Message Passing Architectures Node 0 0, N-1 P Node 1 0, N-1 P Mem $ $ CA CA Interconnect CA CA Mem $ $ P • Cannot directly access memory on another node • IBM SP-2, Intel Paragon • Cluster of workstations P Node 2 0, N-1 © Alvin R. Lebeck 1999 Node 3 0, N-1 10

Important Communication Properties • Bandwidth – – Need high bandwidth in communication Cannot scale, but stay close Limits may be in network, memory, and processor Overhead to communicate is a problem in many machines • Latency – Affects performance, since processor may have to wait – Affects ease of programming, since requires more thought to overlap communication and computation • Latency Hiding – How can a mechanism help hide latency? – Examples: overlap message send with computation, prefetch © Alvin R. Lebeck 1999 11

Small-Scale—Shared Memory • Caches serve to: – Increase bandwidth versus bus/memory – Reduce latency of access – Valuable for both private data and shared data • What about cache coherence? Cache(s) and TLB P P P P $ $ $ $ Main Memory 0 © Alvin R. Lebeck 1999 N-1 12

Cache Coherence Problem (Initial State) P 2 Time P 1 Interconnection Network / Bus x Main Memory © Alvin R. Lebeck 1999 13

Cache Coherence Problem (Step 1) P 2 ld r 2, x Time P 1 Interconnection Network / Bus x Main Memory © Alvin R. Lebeck 1999 14

Cache Coherence Problem (Step 2) P 2 Time P 1 ld r 2, x Interconnection Network / Bus x Main Memory © Alvin R. Lebeck 1999 15

Cache Coherence Problem (Step 3) P 2 Time P 1 ld r 2, x add r 1, r 2, r 4 st x, r 1 Interconnection Network / Bus x Main Memory © Alvin R. Lebeck 1999 16

Time The Problem of Cache Coherence (4) ld r 2, x add r 1, r 3, r 4 st x, r 1 ld r 4, y P 2 P 1 x y ld r 2, x add r 1, r 2, r 3 st y, r 1 ld r 5, x Interconnection Network / Bus x y Main Memory © Alvin R. Lebeck 1999 17

Coherence vs. Consistency • Intuition says loads should return latest value – what is latest? • Coherence concerns only one memory location • Consistency concerns apparent ordering for all locations • A Memory System is Coherent if – can serialize all operations to that location such that, – operations performed by any processor appear in program order » program order = order defined program text or assembly code – value returned by a read is value written by last store to that location © Alvin R. Lebeck 1999 18

Why Coherence != Consistency /* initial A = B = flag = 0 */ P 1 P 2 A = 1; while (flag == 0); /* spin */ B = 1; print A; flag = 1; print B; Intuition says printed A = B = 1 Coherence doesn’t say anything, why? © Alvin R. Lebeck 1999 19

The Sequential Consistency Memory Model sequential processors issue memory ops in program order P 1 P 3 P 2 switch randomly set after each memory op Memory © Alvin R. Lebeck 1999 20

Sufficient Conditions for Sequential Consistency • Every processor issues memory ops in program order • Processor must wait for store to complete before issuing next memory operation • After load, issuing proc waits for load to complete, and store that produced value to complete before issuing next op • Easily implemented with shared bus. © Alvin R. Lebeck 1999 21

Potential Solutions • Snooping Solution (Snoopy Bus): – Send all requests for data to all processors – Processors snoop to see if they have a copy and respond accordingly – Requires broadcast, since caching information is at processors – Works well with bus (natural broadcast medium) – Dominates for small scale machines (most of the market) » probably won’t scale beyond 2 -4 processors • Directory-Based Schemes – – – Keep track of what is being shared in one centralized place Distributed memory => distributed directory (avoids bottlenecks) Send point-to-point requests to processors Scales better than Snoop Actually existed BEFORE Snoop-based schemes © Alvin R. Lebeck 1999 22

Basic Snoopy Protocols • Write Invalidate Protocol: – Multiple readers, single writer – Write to shared data: an invalidate is sent to all caches which snoop and invalidate any copies – Read Miss: » Write-through: memory is always up-to-date » Write-back: snoop in caches to find most recent copy • Write Broadcast (Update) Protocol: – Write to shared data: broadcast on bus, processors snoop, and update copies – Read miss: memory is always up-to-date • Write serialization: bus serializes requests – Bus is single point of arbitration © Alvin R. Lebeck 1999 23

Snoopy Cache-Coherence Protocols • Bus provides serialization point for consistency – but, what about write-buffers? Later in the semester…. • Each cache controller “snoops” all bus transactions – relevant transactions if for a block it contains – take action to ensure coherence » invalidate » update » supply value – depends on state of the block and the protocol • Simultaneous Operation of Independent Controllers © Alvin R. Lebeck 1999 24

Snoopy Design Choices • Controller updates state of blocks in response to processor Processor and snoop events and Cache ld/st generates bus xactions • Often have duplicate cache tags State Tag Data • Snoopy protocol – set of states – state-transition diagram – actions • Basic Choices – write-through vs. write-back Snoop (observed bus transaction) – invalidate vs. update . . . © Alvin R. Lebeck 1999 25

The Simple Invalidate Snoopy Protocol Pr. Rd / Pr. Wr Valid Bus. Wr Pr. Rd / Bus. Rd • Write-through, no-writeallocate cache • Actions: Pr. Rd, Pr. Wr, Bus. Rd, Bus. Wr Invalid Pr. Wr / Bus. Wr © Alvin R. Lebeck 1999 26

A 3 -State Write-Back Invalidation Protocol • 2 -State Protocol + Simple hardware and protocol – Bandwidth (every write goes on bus!) • 3 -State Protocol (MSI) – Modified » one cache has valid/latest copy » memory is stale – Shared » one or more caches have valid copy – Invalid • Must invalidate all other copies before entering modified state • Requires bus transaction (order and invalidate) © Alvin R. Lebeck 1999 27

MSI Processor and Bus Actions • Processor: – Pr. Rd – Pr. Wr – Writeback on replacement of modified block • Bus – Bus Read (Bus. Rd) Read without intent to modify, data could come from memory or another cache – Bus Read-Exclusive (Bus. Rd. X) Read with intent to modify, must invalidate all other caches copies – Writeback (Bus. WB) cache controller puts contents on bus and memory is updated – Definition: cache-to-cache transfer occurs when another cache satisfies Bus. Rd or Bus. Rd. X request • Let’s draw it! © Alvin R. Lebeck 1999 28

MSI State Diagram Pr. Wr Pr. Rd Bus. Rd. X / Bus. WB M Bus. Rd / Bus. WB Pr. Wr / Bus. Rd. X S Pr. Rd Bus. Rd Pr. Rd / Bus. Rd Pr. Wr / Bus. Rd. X © Alvin R. Lebeck 1999 Bus. Rd. X I 29

An example Proc Action P 1 State 1. P 1 read u S 2. P 3 read u S 3. P 3 write u I 4. P 1 read u S 5. P 2 read u S P 2 state ----S P 3 state -S M S S Bus Act Data from Bus. Rd Memory Bus. Rd. X Memory or not Bus. Rd P 3’s cache Bus. Rd Memory • Single writer, multiple reader protocol • Why Modified to Shared? • What if not in any cache? – Read, Write produces 2 bus transactions! © Alvin R. Lebeck 1999 30

4 -State (MESI) Invalidation Protocol • • • Often called the Illinois protocol Modified (dirty) Exclusive (clean unshared) only copy, not dirty Shared Invalid Requires shared signal to detect if other caches have a copy of block • Cache Flush for cache-to-cache transfers – Only one can do it though • What does state diagram look like? © Alvin R. Lebeck 1999 31

4 -State Write-back Update Protocol • Dragon (Xerox PARC) • States – – • • Exclusive (E): one copy, clean, memory is up-to-date Shared-Clean (SC): could be two or more copies, memory unknown Shared-Modified (SM): could be two or more copies, memory stale Modified (M) Adds Bus Update Transaction Adds Cache Controller Update operation Must obtain bus before updating local copy What does state diagram look like? – let’s look at the actions first © Alvin R. Lebeck 1999 32

Basic Snoopy Protocols • Write Invalidate versus Broadcast: – – Invalidate requires one transaction per write-run Invalidate uses spatial locality: one transaction per block Broadcast has lower latency between write and read Broadcast: BW (increased) vs. latency (decreased) tradeoff Name Protocol Type Memory-write policy Write Once Write invalidate Write back after first write Machines using First snoopy protocol. Synapse N+1 Write invalidate Write back 1 st cache-coherent MPs Berkeley Write invalidate Write back Berkeley SPUR Illinois Write invalidate Write back SGI Power and Challenge “Firefly” Write broadcast Write back private, Write through shared SPARCCenter 2000 © Alvin R. Lebeck 1999 33

Larger MPs • • Separate Memory per Processor Local or Remote access via memory controller Cache Coherency solution: non cached pages Alternative: directory/cache that tracks state of every block in every cache – Which caches have a copies of block, dirty vs. clean, . . . • Info per memory block vs. per cache block? – In memory => simpler protocol (centralized/one location) – In memory => directory is ƒ(memory size) vs. ƒ(cache size) • Prevent directory as bottleneck: distribute directory entries with memory, each keeping track of which Procs have copies of their blocks © Alvin R. Lebeck 1999 34

Directory Protocol • Similar to Snoopy Protocol: 3 states – Shared: > 1 processors have data, memory up to date – Uncached – Exclusive: 1 processor(owner) has data; memory out of date • In addition to cache state, must track which processors have data when in shared state • Terms: – Local node is the node where a request originates – Home node is the node where the memory location of an address resides – Remote node is the node that has a copy of a cache block, whether exclusive or shared. © Alvin R. Lebeck 1999 35

Example Directory Protocol • Message sent to directory causes 2 actions: – update the directory – more messages to satifty request • Block is in Uncached state: the copy in memory is the current value, & only possible requests for that block are: – Read miss: requesting processor is sent back the data from memory and the requestor is the only sharing node. The state of the block is made Shared. – Write miss: requesting processor is sent the value and becomes the Sharing node. The block is made Exclusive to indicate that the only valid copy is cached. Sharers indicates the identity of the owner. • Block is Shared, the memory value is up-to-date: – Read miss: requesting processor is sent back the data from memory & requesting processor is added to the sharing set. – Write miss: requesting processor is sent the value. All processors in the set Sharers are sent invalidate messages, & Sharers set is to identity of requesting processor. The state of the block is made Exclusive. © Alvin R. Lebeck 1999 36

Example Directory Protocol • Block is Exclusive: current value of the block is held in the cache of the processor identified by the set Sharers (the owner), & 3 possible directory requests: – Read miss: owner processor is sent a data fetch message, which causes state of block in owner’s cache to transition to Shared and causes owner to send data to directory, where it is written to memory and sent back to the requesting processor. Identity of requesting processor is added to set Sharers, which still contains the identity of the processor that was the owner (since it still has a readable copy). – Data write-back: owner processor is replacing the block and hence must write it back. This makes the memory copy up-to-date (the home directory essentially becomes the owner), the block is now uncached, and the Sharer set is empty. – Write miss: block has a new owner. A message is sent to old owner causing the cache to send the value of the block to the directory from which it is send to the requesting processor, which becomes the new owner. Sharers is set to identity of new owner, and state of block is made Exclusive. © Alvin R. Lebeck 1999 37