Caches prepared and Instructed by Shmuel Wimer Eng

Caches prepared and Instructed by Shmuel Wimer Eng. Faculty, Bar-Ilan University Cashes 1

Amdahl’s Law Speedup: How much faster a task will run on the computer with an enhancement, compared to the original computer. Cashes 2

Principle of Locality • Temporal locality (locality in time): If an item is referenced, it will tend to be referenced again soon. • Spatial locality (locality in space): If an item is referenced, items whose addresses are close will tend to be referenced soon. • locality in programs – loops - temporal – instructions are usually accessed sequentially - spatial – Data access of array - spatial Cashes 3

Memory Hierarchy Cashes 4

Memory Hierarchy • The memory system is organized as a hierarchy – A level closer to the processor is a subset of any level further away. – All the data is stored at the lowest level. • Hierarchical implementation makes the illusion of a memory size as the largest, but can be accessed as the fastest. Cashes 5

Hit and Miss • In a pair of levels one is upper and one is lower. • The unit within each level is called a block. • We transfer an entire block when we copy something between levels. Hit rate, or hit ratio, is the fraction of memory accesses found in the upper level. Miss rate = 1 – hit rate. Cashes 6

Hit time: the time required to access a level of the memory hierarchy. • Includes the time needed to determine whether hit or miss. Miss penalty: the time required to fetch a block into the memory hierarchy from the lower level. • Includes the time to access the block, transmit it from the lower level, and insert it in the upper level. The memory system affects many other aspects of a computer: • How the operating system manages memory and I/O • How compilers generate code • How applications use the computer Cashes 7

This structure allows the processor to have an access time that is determined primarily by level 1 of the hierarchy and yet have a memory as large as level n. Cashes 8

General Memory architecture Bit-line Conditioning Word-lines Bit-lines Row Decoder 2 n-k Array of 2 nx 2 m cells, organized in 2 n-k rows by 2 m+k columns k 1 n 3 Column Decoder 8 bits Column Circuitry 2 m bits m=2 4 -word by 8 -bit folded memory 9

6 -Transistor SRAM Cell February 2014 Cashes 10

Dynamic RAM bit 0 bit 1 bit 511 word 0 word 1 word 255 Cashes 11

Lithography simulation Layout design Silicon Cashes 12

Polysilicon Word-Line Sense Amp Metal Bit-Line n+ Diffusion Word-Line Decoder Bit-Line Contact Capacitor Word-Line Decoder Sense Amp 13

Sum-addressed Decoders 14

15

We can derive the equations of the carries from the required and generated carries below. 0 0 0 0 1 1 0 1 1 0 0 1 1 0 0 1 1 1 16

17

18

August 2010 19

20

Below is a comparison of sum-addressed decoder with ordinary decoder combined with a ripple carry adder (RCA) and carry look ahead adder (CLA). A significant delay and area improvement is achieved. 21

Requesting data from the cache Two questions : • How do we know if a data item is in the cache? • If it is, how do we find it? Cashes 22

Direct-Mapped Cache Each memory location is mapped to one cache location Mapping between addresses and cache locations: (Block address in Mem) % (# of blocks in cache) Modulo is computed by using log 2(cache size in blocks) LSBs of the address. The cache is accessed directly with the LSBs of the requested memory address. Problem: this is a many-to-one mapping. A tag field in a table containing the MSBs to identify whether the block in the hierarchy corresponds to a requested word. Cashes 23

Mem address mod 8 = 101 Mem address mod 8 = 001 tag Cashes 24

Cashes 25

Some of the cache entries may still be empty. We need to know that the tag should be ignored for such entries. We add a valid bit to indicate whether an entry contains a valid address. Cashes 26

Cache Access Sequence Cashes 27

Referenced address is divided into • a cache index, used to select the block • a tag field, compared with the value of the tag field of the cache Cashes 28

Cache Size Cashes 29

Example: How many total bits are required for a direct-mapped cache with 16 KB of data and 4 -word blocks, assuming a 32 -bit address? Each block has 4 x 32 = 128 bits of data, plus a tag of 32 - 10 - 2 2 bits, plus a valid bit. The total cache size is therefore For a 16 KB cache it is about 1. 15 times as many as needed just for data storage. Cashes 30

Cashes 31

Block Size Implications • Larger blocks exploit spatial locality to lower miss rates. • Block increase will eventually increase miss rate • Spatial locality among the words in a block decreases with a very large block. – The number of blocks held in the cache will become small. – There will be a big competition for these blocks. – A block will be thrown out of the cache before most of its words are accessed. Cashes 32

Miss rate versus block size Cashes 33

A more serious issue in block size increase is the increase of miss cost. • Determined by the time required to fetch the block and load it into the cache. Fetch time has two parts: • the latency to the first word, and • the transfer time for the rest of the block. Transfer time (miss penalty) increases as the block size grows. The increase in the miss penalty overwhelms the decrease in the miss rate for large blocks, thus decreasing cache performance. Cashes 34

• Shortening transfer time is possible by early restart, resuming execution once the word is returned. – Useful for instruction, that are largely sequential. – Requires that the memory delivers a word per cycle. – Less effective for data caches. High probability that a word from different block will be requested soon. – If the processor cannot access the data cache because a transfer is ongoing, it must stall. • Requested word first – starting with the address of the requested word and wrapping around. – Slightly faster than early restart. Cashes 35

Handling Cache Misses Modifying the control of a processor to handle a hit is simple. Misses require extra work done with the processor’s control unit and a separate controller. Cache miss creates a stall by freezing the contents of the pipeline and programmer-visible registers, while waiting for memory. Cashes 36

Steps taken on an instruction cache miss: 1. Send to the memory the original PC value. 2. Instruct main memory to perform a read and wait for the memory to complete its access. 3. Write the cache entry: memory’s data in the entry’s data portion, upper bits of the address into the tag field, turn the valid bit on. 4. Restart the instruction execution at the first step, which will re-fetch the instruction, this time finding it in the cache. The control of the data cache is similar: miss stalls the processor until the memory responds with the data. Cashes 37

Handling Writes After a hit writes into the cache, memory has a different value than the cache. Memory is inconsistent. We can always write the data into both the memory and the cache, a scheme called write-through. Write miss first fetches block from memory. After it is placed into cache, we overwrite the word that caused the miss into the cache block and also write it to the main memory. Write-through is simple but has bad performance. Write is done both to cache and memory, taking many clock cycles (e. g. 100). If 10% of the instructions are stores and the CPI without misses was 1. 0, new CPI is 1. 0 + 100 x 10% = 11, a 10 x slowdown! Cashes 38

Speeding Up A write buffer is a queue holding data waiting to be written to memory, so the processor can continue working. When a write to memory completes, the entry in the queue is freed. If the queue is full when the processor reaches a write, it must stall until there is an empty position in the queue. An alternative to write-through is write-back. At write, the new value is written only to the cache. The modified block is written to the main memory when it is replaced. Write-back improves performance when processor generates writes faster than the writes can be handled by main memory. Implementation is more complex than write-through. Cashes 39

Cache Example (Data and Instruction) Miss sends the address to memory. Returned data is written into the cache and is then read to fulfill request. instruction cache: from PC data cache: from ALU 18 Hit selects by offset the word from the block Cashes 40

Main Memory Design Considerations Cache misses are satisfied from DRAM main memory, designed for density rather than access time. Miss penalty can be reduced by increasing bandwidth from the memory to the cache. Bus clock rate is 10 x slower than processor, affecting the miss penalty. Assume • • • 1 memory bus clock cycle to send the address 15 memory bus clock cycles for each DRAM access initiated 1 memory bus clock cycle to send a word of data Cashes 41

Cashes 42

Cache Performance Two techniques to reduce miss rate: • Reducing the probability that two different memory blocks will contend for the same cache location by associativity. • Adding a level to the hierarchy, called multilevel caching. Cashes 43

CPU Time CPU time = (CPU execution clock cycles + Memorystall clock cycles) x Clock cycle time Memory-stall clock cycles = Read-stall cycles + Writestall cycles Read-stall cycles = Reads/Program x Read miss rate x Read miss penalty Write-stall cycles = Writes/Program x Write miss rate x Write miss penalty + Write buffer stall cycles (writethrough) Write buffer term is complex. It can be ignored for buffer depth > 4 words, and a memory capable of accepting writes at > 2 x rate than the average write frequency. Cashes 44

Write-back also has additional stalls arising from the need to write a cache block back to memory when it is replaced. Write-through has about the same read and write miss penalties (fetch time of block from memory). Ignoring the write buffer stalls, the miss penalty is: Memory-stall clock cycles (simplified) = Memory accesses/Program x Miss rate x Miss penalty = Instructions/Program x Misses/Instruction x Miss penalty Cashes 45

Cashes 46

Example: Accelerating processor but not memory. Memory stalls time fraction is increased. Cashes 47

Relative cache penalties increase as a processor becomes faster. If a processor improves both CPI and clock rate • The smaller the CPI, the more impact of stall cycles is. • If the main memories of two processors have the same absolute access times, higher processor’s clock rate leads to larger miss penalty. The importance of cache performance for processors with small CPI and faster clock is greater. Cashes 48

Reducing Cache Misses Direct map scheme places a block in a unique location. Fully associative scheme places a block in any location. • All cache’s entries must be searched. • Expensive: done in parallel with a comparator for each entry. • Practical only for caches with a small number of blocks. Cashes 49

Cashes 50

Cache size (blocks) = number of sets x associativity. For fixed cache size, increasing the associativity decreases the number of sets. Cashes 51

Example: Misses and associativity in caches. Three caches of 4 1 -word blocks, fully associative, two-way set associative, and direct mapped. For the sequence of block addresses: 0, 8, 0, 6, 8, what is the number of misses for each cache? direct mapped 5 misses Cashes 52

two-way set associative 4 misses 3 misses fully associative Cashes 53

Size and associativity are dependent in determining cache performance. For 8 blocks in the cache, there are no replacements in the two -way set-associative cache. (why? ) There are same number of misses as the fully associative cache. For 16 blocks, all three caches would have the same number of misses. Benchmarks of a 64 KB data cache with a 16 -word block Cashes 54

Four-way set-associative cache 1 -word block 4 -block set parallel MUX with a decoded select signal Cashes 55

Locating a Block in the Cache Set is found by the index. Tag of a block within the appropriate set is checked for matching. Block offset is the address of the word within the block. For speed all the tags in a set are searched in parallel. In a fully associative cache, we search the entire cache without any indexing. Huge HW overhead. The choice among direct-mapped, set-associative, or fully associative depends on the miss (performance) cost versus HW cost (power, area). Cashes 56

Example: Size of tags versus set associativity Given cache of 4 K=212 blocks, a 4 -word block size, and a 32 -bit address. What is the total number of tag bits? There are 16=24 bytes / block. 32 -bit address yields 32 -4 =28 bits for index and tag. Direct-mapped cache has 12=log 2(4 K) bits of index. Tag is 28 -12=16 bits, yielding a total of 16 x 4 K = 64 Kbits of tags. Cashes 57

For a 2 -way set-associative cache, there are 2 K = 211 sets, and the total number of tag bits is (28 - 11) x 2 K =34 x 2 K = 68 Kbits. For a 4 -way set-associative cache, there are 1 K = 210 sets, and the total number of tag bits is (28 - 10) x 4 x 1 K = 72 Kbits. Fully associative cache has one set with 4 K blocks, and the total number of tag bits is 28 x 4 K x 1 = 112 K bits. Cashes 58

Which Block to Replace? In a direct-mapped cache the requested block can go in exactly one position. In a set-associative cache, we must choose among the blocks in the selected set. The most commonly used scheme is least recently used (LRU), where the block replaced is the one that has been unused for the longest time. For a two-way set-associative cache, tracking when the two elements were used can be implemented by keeping a single bit in each set. Cashes 59

As associativity increases, implementing LRU gets harder. Random • • • Spreads allocation uniformly. Blocks are randomly selected. System generates pseudorandom block numbers to get reproducible behavior (useful for HW debug). First in, first out (FIFO) Because LRU can be complicated to calculate, this approximates LRU by determining the oldest block rather than the LRU. Cashes 60

Multilevel Caches Used to reduce miss penalty. Many processors support an on-die 2 nd-level (L 2) cache. L 2 is accessed whenever a miss occurs in L 1. If L 2 contains the desired data, the miss penalty for L 1 is the access time of L 2, much less than the access time of main memory. If neither L 1 nor L 2 contains the data, main memory access is required, and higher miss penalty incurs. Cashes 61

Example: performance of multilevel caches Given a 5 GHz processor with a base CPI of 1. 0 if all references hit in the L 1. Main memory access time is 100 ns, including all the miss handling. L 1 miss rate per instruction is 2%. How faster the processor is if we add a L 2 that has a 5 ns access time for either a hit or a miss, reducing the miss rate to main memory to 0. 5% ? Cashes 62

Miss penalty to main memory (memory-stall): 5 GHz x 100 ns = 500 cycles. The effective CPI with L 1: Base CPI + Memory-stall cycles per instruction = 1 + 500 x 2% = 11 The effective CPI with L 2: 1 + 25 x (2% - 0. 5%) + (500 + 25) x 0. 5% = 4 The processor with L 2 is faster by: 11 / 4 = 2. 8 Cashes 63

מה מספר מחזורי השעון המרבי שעשוי להידרש בעת גישה לזיכרון ? הראשי? מהו רצף האירועים המתרחש במצב קיצוני כזה Maximum clock cycles occur when L 1 missed first, then L 2 missed, then write-back takes place. L 2 access cycles: (20 n. Sec) / (2 n. Sec) = 10 cycles Main memory access cycles: (0. 2 µSec) / (2 n. Sec) = 100 cycles Block is 32 Bytes and memory bus is 128 bits (16 Bytes), two bus transactions of 16 Bytes each are required. The first 16 bytes take 100 cycles, the next 16 bytes takes one cycle. Getting a new block from the memory may evict a block from L 2, which is a write-back. In that case the evicted block must be written into the memory, requiring a total of L 2 -memory writeback 2 x (100 + 1) = 202 cycles. Febuary 2014 Cashes 66

Summing all L 1 miss + L 2 miss + write-back = 1 + 10 + 202 = 213 cycles ( כולל פקודות AMAT) מהו מספר מחזורי השעון הממוצע בגישה לזיכרון ? ונתונים The weight of instruction accesses to memory is 1/(1 + 0. 4), while the weight of data accesses is 0. 4/(1 + 0. 4). Therefore AMATtotal = 1/1. 4 AMATinst + 0. 4/1. 4 AMATdata For any 2 -level cache system there is AMAT = (L 1 hit time) + (L 1 miss rate) x (L 2 miss rate) x (main memory transfer time). AMAT must account for the average percentage of L 2 dirty blocks, which for the given L 2 means that 50% of the blocks must be updated in main memory upon L 2 miss, yielding a factor of 1. 5 multiplying (100 +1). Febuary 2014 Cashes 67

AMATinst = 1 + 0. 02 x 10 + 0. 02 x 0. 1 x 1. 5 x (100 + 1) = 1. 503 AMATdata = 1 + 0. 15 x 10 + 0. 15 x 0. 1 x 1. 5 x (100 + 1) = 4. 7725 AMATtotal = 1/1. 4 x 1. 503 + 0. 4/1. 4 x 4. 7725 = 2. 44 Febuary 2014 Cashes 68

Victim Cache Memory hierarchy so far was inclusive. Exclusive organization of caches may help to overcome the associativity vs. block-size tradeoff. A direct-mapped/limited-associativity cache + small fully associative victim cache can perform as a multiway set associative. A block is found either in the main cache or in the victim cache but not in both. Proposed in 1990’s for L 1. Used today at higher levels L 3/L 4 by Intel and IBM (Power. PC). March 2020 Cashes 69

Victim cache captures temporal locality: blocks accessed frequently, even if replaced from the main cache, remain within the larger cache organization. It addresses the limitation of main cache by giving blocks a second chance if they exhibit conflict with other blocks competing for the same set. The victim cache contains the data items that have been thrown out of the main cache. Cache probe looks in both caches. If the requested block is found in the victim, that block is swapped with the block it replaces in main cache. March 2020 Cashes 70

March 2020 Cashes 71

FIFO replacement policy of victim cache effectively achieves true-LRU behavior (why? ). A reference to the victim cache pulls the referenced block out of it. Thus, its LRU block is by definition the oldest one there. Let b 1 be thrown by FIFO (victim oldest ) and b 2 reside in the victim but used last time before b 1 did (LRU). At its usage time b 1 was pulled into L 1, so its reentrance into the victim occurred when b 2 was already there, hence b 1 cannot be older than b 2. March 2020 Cashes 72

Assist Cache Was motivated by the processor’s stride-based prefetch. If too aggressive it would prefetch data that would victimize soon-to-be-used cache entries. The incoming prefetch block is loaded into the associative, promoted into the main only if it exhibits temporal locality by being referenced again soon. Blocks exhibiting only spatial locality are not moved to the main cache and are replaced to main memory in FIFO. Used by HP in 1990’s in HP-7200 microprocessor. March 2020 Cashes 73

March 2020 Cashes 74

An incoming block is moved into the assist cache; the replaced block is thrown out. If a reference misses the main and hits the assist, the assist’s block is swapped into the main. The corresponding main’s block is inserted into the assist. Only blocks exhibiting temporal locality get promoted to the main cache, hence separating blocks exhibiting spatial locality from those exhibiting temporal locality. Using FIFO, the scheme may discard blocks exhibiting streaming behavior once used, without moving them into the main cache. March 2020 Cashes 75

Dynamic L 1 Exclusion Consider direct-mapped L 1 and a loop-based code, where instructions conflict with each other (e. g. by instructions external of the loop). We like to stick the loop instruction to L 1 and avoid its evection, as it will probably be required again soon. To address conflicts without disturbing ordinary L 1 behavior, a sticky bit at each entry indicates that its current block is valuable and should not be replaced. A hit-last bit at each cache block indicates that it was used (again) the last time it was already in the cache. Cashes 76

March 2020 Cashes 77

A, B: cached blocks (same index) a, b: requested blocks ha’ hb : hit-last bit of the incoming block action condition March 2020 Cashes 78

Homework: Let L 1 block be 1 -word and instructions a and b mapped to same index. Assume initial state A, s and ha = hb = 0 in L 2. Find the miss rate in ordinary and a dynamic exclusion L 1 in the following codes. March 2020 Cashes 79

The scheme extends to data caches as well. Any block in the backing store is potentially cacheable or non-cacheable. The partition changes dynamically with application behavior. March 2020 Cashes 80

Summary – Four Questions Q 1: Where can a block be placed in the upper level? (block placement) Q 2: How is a block found if it is in the upper level? (block identification) Q 3: Which block should be replaced on a miss? (block replacement) Q 4: What happens on a write? (write strategy) Cashes 81