Chapter 4 Cache Memory Computer memory is organized

Chapter 4 Cache (高速缓存) Memory § Computer memory is organized into a hierarchy (层次结构或存储体系). At the highest level(closest to the processor) are the processor registers. Next come a cache or two levels of cache, denoted(表示) the L 1 and L 2 cache. Next comes main memory, which is usually made out of dynamic(动态) random-access memory(DRAM). All of these are considered internal to the computer system. The hierarchy continues with external memory, with the next level typically being a fixed hard disk, and one or more levels below that consisting of removable(可更换) media(介质) such as ZIP cartridges, optical disks, and tape.

Chapter 4 Internal Memory § As one goes down the memory hierarchy (层次结构或存储体系), one finds decreasing cost/bit, increasing capacity(容量), and slower access time. It would be nice to use only the fastest memory, but because that is the most expensive memory, we trade off (牺牲)access time for cost by using more of the slower memory. The trick(诀窍) is to organize the data and programs in memory so that the memory words needed are usually in the faster memory.

Chapter 4 Internal Memory § In general, it is likely that most future accesses(存取) to main memory by the processor will be to locations(单元) recently accessed. So the cache automatically retains(保留) a copy of some of the recently used words from the DRAM. If the cache is designed properly(适当地), then most of the time the processor will request memory words that are already in the cache.

4. 1 Computer Memory System Overview Characteristics (特性) § Location(Position) • Processor(register) • Internal (main, cache)(directly accessible(可存取的) by the CPU) • External(secondary or auxiliary)(accessible by the CPU via an I/O module(模块) or controller)

§ Capacity(容量): typically expressed in terms of bytes(字节) or words(字) for internal memory and in terms of bytes for external memory • Word(字) size: The “natural” unit(单位) of organization(组织) of memory. The size of the word is typically equal to the number of bits used to represent( 表示) a number and to the instruction( 指令) length.

• Addressable(可寻址的) units: Smallest location which can be uniquely(唯一) addressed. In many systems, the addressable unit is the word(字). However, some systems allow addressing at the byte level. In any case, the relationship(关系) between the length in bits A of an address and the number N of addressable units is 2 A=N

Characteristics § Unit of transfer(传输): For main memory, this is the number of bits read out of or written into memory at a time. The unit of transfer need not equal a word or an addressable unit. It is usually governed(controlled) by data bus width. For external memory, data are often transferred in much larger units than a word, and these are referred to as (称作 为) blocks(块).

Access method(存取方法) n Sequential(顺序) access: Memory is organized into units of data, called records(记录). Access must be made in a specific linear (线性的) sequence. Stored addressing(寻址) information is used to separate(分开的) records and assist in the retrieval(检索) process(过程). A shared(共享的) read/write mechanism(机构) is used, and this must be moved from its current location to the desired(指定的) location, passing and rejecting(忽略) each intermediate(中间的) record. Thus, the time to access an arbitrary(任 意的) record is highly variable(可变的). Tape units , discussed in Chapter 6, are sequential access.

Access method(存取方法) • Direct access: As with sequential access, direct access involves(涉及) a shared read-write mechanism. However, individual(各个) blocks or records have a unique address based on physical location. Access is accomplished(完成) by direct access to reach a general vicinity(附近) plus sequential searching, counting(记数), or waiting to reach the final location. Again, access time is variable(可变的). Disk units are direct access.

Access method(存取方法) § Random(随机) access: Each addressable location in memory has a unique, physically wired-in(固定的/编排好的) addressing mechanism(机构). The time to access a given location is independent of the sequence(顺序) of prior(先前的) accesses and is constant(恒定的). Thus, any location can be selected at random and directly addressed and accessed. Main memory and some cache systems are random access.

Access method(存取方法) § Associative(相联): This is a random-access type of memory that enables one to make a comparison(比较) of desired(指定的) bit locations within a word for a specified match(匹配), and to do this for all words simultaneously(同时). Thus, a word is retrieved(检索) based on a portion of its contents(内容) rather than its address. As with ordinary random-access memory, each location has its own addressing mechanism, and retrieval time is constant(恒定) independent of location or prior access patterns(样式). Cache memories may employ(使用) associative access.

Performance(性能) § Access time(存取时间): For RAM, this is the time it takes to perform a read or write operation: that is, the time from the instant(时刻) that an address is presented( 提交) to the memory to the instant that data have been stored or made available for use. For non-RAM, access time is the time it takes to position(定位) the readwrite mechanism at the desired(指定的/期望 的) location.

Performance(性能) • Memory cycle time(周期时间): This concept is primarily applied to RAM and consists of the access time plus any additional time required before a second access can commence(开始). This additional time may be required for transients (过渡) to die out(消失) on signal lines or to regenerate(再生) data if they are read destructively(破 坏性).

Performance(性能) § Transfer rate(传输率): This is the rate at which data can be transferred into or out of a memory unit. For RAM, it is equal to 1/(cycle time). For non-RAM, the following relationship holds: N TN=TA+ ─ R where TN=Average time to read or write N bits TA=Average(平均的) access time N=Number of bits(位数) R=Transfer rate, in bits per second(bps)

Characteristics § Physical Types • Semiconductor(半导体) ²RAM • Magnetic Surface(表面) memory ²Disk & Tape • Optical(光) ²CD & DVD • Magneto-Optical(磁光)

Characteristics(特性) § Physical characteristics • Volatile(易失) memory: Information decays(消失 ) naturally or is lost when electrical power(电 源) is switched off • Non-volatile memory: Information once recorded remains without deterioration(退化) until deliberately(有意地) changed. • Nonerasable(非擦除) memory: Information cannot altered(改变), except by destroying(破 坏) the storage. § Organization(组织): Physical arrangement of bits into words.

Memory Hierarchy(层次结构) § The design constraints(限制) on a computer’s memory can be summed up by three questions: How much(Capacity)? How fast(Access time)? How expensive(Cost)? § At any given time, a variety of technologies are used to implement(实现) memory systems. Across(遍及) this spectrum(范围) of technologies, the following relationships(关系 ) hold: • Faster access time, greater cost per bit • Greater capacity, smaller cost per bit • Greater capacity, slower access time

Memory Hierarchy (层次结构) § The way out of this dilemma(困境) is not to rely on(依靠) a single memory component or technology, but to employ a memory hierarchy. A typical hierarchy is illustrated(展示) in figure 4. 1.

a. Decreasing cost per bit b. Increasing capacity c. Increasing access time d. Decreasing frequency( 次数) of access of the memory by the CPU Write-Once Read-Many Figure 4. 1 The Memory Hierarchy

Figure 4. 2 Performance of a Simple Two-Level Memory Hit ratio (命中率)

locality of reference(局部访问原理) § The basis for the validity of condition (d) is a principle known as locality of reference. During the course of execution of a program, memory references by the processor, for both instructions and data, tend to cluster(聚集). § Programs typically contain a number of iterative(迭代) loops(循环) and subroutines(子程序 ). Once a loop or subroutine is entered(进入), there are repeated(重复) references to a small set of instructions. Similarly, operations on tables(表) and arrays(数组) involve(涉及) access to a clustered set of data words. Over a long period of time, the clusters in use change, but over a short period of time, the processor is primarily working with fixed clusters of memory references.

locality of reference(局部访问原理) § Accordingly, it is possible to organize data across the hierarchy such that the percentage( 百分比) of accesses(存取) to each successively (依 次地) lower level is substantially(显著地) less than that of the level above. Consider the two-level example already presented. Let level 2 memory contain all program instructions and data. The current clusters(簇) can be temporarily placed in level 1. From time to time, one of the clusters in level 1 will have to be swapped(交换) back to level 2 to make room for a new cluster coming in to level 1. On average, however, most references(访问) will be to instructions and data contained in level 1.

4. 2 Cache (高速缓冲) Memory Principles § Cache memory is intended to give memory speed approaching(接近) that of the fastest memories available, and at the same time provide a large memory size at the price of less expensive types of semiconductor(半导体) memories. § There is a relatively large and slow main memory together with a smaller, faster cache memory. The cache contains a copy of portions (部分) of main memory.

4. 2 Cache (高速缓冲) Memory § When the CPU attempts to read a word of memory, a check is made to determine if the word is in the cache. If so, the word is delivered( 传输) to the CPU. if not, a block of main memory, consisting of some fixed number of words, is read into the cache and then the word is delivered to the CPU. § Because of the locality of reference, when a block of data is fetched into the cache to satisfy(满足) a single memory of reference(访问), it is likely that future references will be to other words in the block. Block Transfer

Cache/Main Memory structure(结构) 行 标记 块 M=2 n/K C<<M Fig. 4. 4 Cache/Main Memory structure

Fig. 4. 5 Read Operation

Typical Cache Organization(组织) Bus Main (缓冲器) Bus Fig. 4. 6 Typical Cache Organization Memory

4. 3 Elements(要素) of Cache Design § Size(容量) § Mapping(映射) Function • Direct (直接) • Associative (关联) • Set Associative(组关联) § Replacement Algorithm(算 法) • Least recently used(LRU) (近期最少使用) • First in first out(FIFO) • Least frequently used(LFU) (最不经常使用) • Random (随机) § Write policy(写策略) • Writ through(通写) • Write back(回写) • Write once § Line Size(行大小) § Number of caches • Single or two level • Unified or split(统一/分 开) Table 4. 2 Elements of Cache Design

Cache Size(容量/大小) § Small enough so that the overall average cost per bit is close to that of main memory alone and large enough so that the overall average access time is close to that of the cache alone. § Other motivations(动机/目的) for minimizing (to make sth small ) cache size. • The larger the cache, the larger the number of gates(门电 路) involved in addressing the cache. The result is that larger caches tend to be slightly slower than small ones. § The cache sizes of between 1 K and 512 K words will be effective. Because the performance of the cache is very sensitive(敏感的) to the nature of the workload( 作负荷), it is impossible to arrive at a single “optimum”(最优) cache size.

Mapping Function(功能/函数) § Fewer cache lines than main memory blocks, an algorithm (算法) is needed for mapping main memory blocks into cache lines. Further, a means is needed for determining which main memory block currently occupies (stores) a cache line. § The choice of the mapping function(direct, associative, and set associative) dictates(支配) how the cache is organized. § In all three cases, we will use the example including the following elements: • Cache of 64 k. Byte • Cache block of 4 bytes ² i. e. cache is 16 k (214) lines of 4 bytes • 16 MBytes main memory • 24 bit address ² (224=16 M), 4 M blocks of 4 bytes each

Direct Mapping (直接映射) § Each block of main memory maps (映射) to only one cache line • i. e. if a block is in cache, it must be in one specific place § Formula(公式) to map a memory block to a cache line: I=J modulo m where I = cache line umber(行号) J = main memory block number(块号) m=number of lines in the cache(行数) § Shrinking(缩小) our example to a cache size of 4 lines (each line/block still contains 4 words): • Cache Line ² 0 ² 1 ² 2 ² 3 Memory Block Held 0, 1, 2, 3, 4, 5, 6, 7, 8, 12 9, 13 10, 14 11, 15

Direct Mapping with C=4 Block 0 Line 0 Block 1 Line 1 Block 2 Line 2 2 Block 3 Line 3 Block 4 Cache Memory Block 5 Block 6 Main Memory Block 7

Direct Mapping(直接映射) § The mapping function is easily implemented using the address. § For purposes of cache access, each main memory address can be viewed as consisting of three fields. • The least significant(最低有效位) w bits identify a unique word or byte within a block of main memory • The remaining(其余) s bits specify one of the 2 s blocks of main memory. • The cache logic interprets these s bits as a tag(标记) of s – r bits(most significant portion) and a line field of r bits identifying(确定) one of the m=2 r lines of the cache. Tag s–r Line r Word w

Direct Mapping Cache Line Table The effect(结果) of this mapping is that blocks of main memory are assigned(分配/指定) to lines of the cache as follows: Cache line Main memory blocks assigned 0 0, m, 2 m, …, 2 s - m 1 1, m +1, 2 m + 1, …, 2 s –m +1 . . . m-1 m – 1, 2 m-1, 3 m – 1, …, 2 s -1

Direct Mapping Example Cache line Starting memory address of block 0 000000, 010000, …, FF 0000 1 000004, 010004, …, FF 0004 . . . M-1 00 FFFC, 01 FFFC, …, FF FFFC 16 -K Lines Cache m=16 K=214 and i=j modulo 214 Memory Address = 8 Tag 14 Line 2 Word

Direct Mapping Cache Organization

Direct Mapping Features(特点) § Simple § Inexpensive § Fixed location for any given block If a program happens to reference words repeatedly from two different blocks that map into the same line, then the blocks will be continually(频繁地) swapped(交换) in the cache and the hit ratio will be low

Fully Associative(全关联) Mapping § A main memory block can load into any line of cache § Memory address is interpreted as tag and word § Tag uniquely identifies block of memory § Every line’s tag is examined for a match § Cache searching gets expensive Block 0 Line 1 Line 2 Block 1 Block 2 Block 3 Block 4 Line 3 Block 5 Block 6 Block 7

Fully Associative Cache Organization

Associative Mapping Example A main memory address consists of a 22 -bit tag and a 2 -bit byte number. The 22 -bit tag must be stored with the 32 -bit block of data for each line in the cache. Compare tag field with tag entry in cache to check for hit(命中). Least significant 2 bits of address identify which word is required from 4 -word block Tag Word 22 Main Memory Address 2

Associative Mapping Features § Flexibility as to which block to replace when a new block is read into the cache § Replacement algorithms(算法), discussed later in this section, are designed to maximize the hit ratio(命中率) § The complex circuitry(电路) required to examine the tags of all cache lines in parallel.

Set Associative(组关联) Mapping § Cache is divided into v sets(组), each of which consists of k lines. The relationships are Where m=v×k i = j modulo v i = cache set number j = main memory block number m = number of lines in the cache This is referred to as k-way(路) set associative mapping. With set associative mapping, block Bj can be mapped into any of the lines of set i. The cache control logic interprets a memory address simply as three fields: tag, set, and word. The d set bits specify one of v = 2 d sets. The s bits of the tag and set fields specify one of the 2 s blocks of main memory.

K- Way Set Associative Mapping With fully associative mapping, the tag in a memory address is quite large and must be compared to the tag of every line in the cache. With k-way set associative mapping, the tag in a memory address is much smaller and is only compared to the k tags within a single set.

Two Way Set Associative Mapping The 13 -bit set number identifies a unique set of two lines within the cache. It also gives the number of the block in main memory, modulo 213. Note that no two blocks that map into the same cache set have the same tag number. For reading, the 13 -bit set number is used to determine which set of two lines is to be examined. Both lines in the set are examined for a match with the tag number of the address to be accessed. 9 Tag 13 Set Main Memory Address 2 Word

Set Associative Mapping § V=m, k=1, the set associative mapping reduces to direct mapping § V=1, k=m, it reduces to fully associative mapping. § V=m/2, k=2 is the most common set associative mapping. It significantly improves the hit ratio over direct mapping § V=m/4, k=4 makes a modest additional improvement for a relatively small additional cost. § Further increases in the number of lines per set have little effect.

Replacement Algorithms(替换算法) Direct mapping: only one possible line for any particular block, and no choice Associative and set associative mapping : algorithms needed and implemented in hardware.

Replacement Algorithms (替换算法) • LRU(least recently used近期最少使用): Replace that block in the set that has been in the cache longest with no reference to it. • FIFO(first-in-first-out): Replace that block in the set that has been in the cache longest. FIFO is easily implemented as a round-robin or circular buffer technique. • LFU(least frequently used最不经常使用): Replace that block in the set that has experienced the fewest references. LFU could be implemented by associating a counter with each line • Random : Not based on usage and pick a line at random from among the candidate lines.

Write Policy(写策略) § The line altered in the cache must be written back into main memory before replacing it. A variety of write policies , with performance and economic trade-offs, are possible. § Two problems to contend with: • More than one device may have access to main memory. For example, an I/O module may be able to read/write directly to memory. If a word has been altered only in the cache, then the corresponding memory word is invalid. Further , if an I/O device has altered main memory, then the cache word is invalid.

Write Policy(写策略) • Multiple processors are attached(连接) to the same bus and each processor has its own local cache. Then , if a word is altered(改变) in one cache, it could conceivably invalidate(使 无效) a word in other caches.

Write Policy(写策略) § Write through: all write operations are made to main memory as well as to the cache, ensuring(保证) that main memory is always valid. Any other processorcache module can monitor(监视) traffic to main memory to maintain(维持) consistency(一致性) within its own cache. § The main disadvantage of this technique is that it generates substantial memory traffic and may create a bottleneck.

Write Policy § Write back: updates are made only in the cache. When an update occurs, an UPDATE bit associated with the line is set. Then, when a block is replaced, it is written back to main memory if and only if the UPDATE bit is set. § The problem with write back is that portions of main memory are invalid, and hence accesses by I/O modules can be allowed only through the cache. This makes for complex circuitry and a potential bottleneck.

Write Policy § In a bus organization in which more than one device has a cache and main memory is shared, a new problem is introduced. If data in one cache is altered, this invalidates not only the corresponding word in main memory, but also that same word in other caches(if any other cache happens to have that same word). Even if a write-through policy is used, the other caches may contain invalid data.

Write Policy § A system that prevents this problem is said to maintain cache coherency(一致性). § Possible approaches to cache coherency include the following: • Bus watching(监测) with write through • Hardware transparency(透明性) • Noncachable memory

Write Policy § Bus watching with write through: each cache controller monitors the address lines to detect write operations to memory by other bus masters. If another master writes to a location in shared memory that also resides in the cache memory, the cache controller invalidates that cache entry(项). This strategy depends on the use of a writethrough policy by all cache controllers. § Hardware transparency: Additional hardware is used to ensure that all updates to main memory via cache are reflected in all caches. Thus, if one processor modifies a word in its cache, this update is written to main memory. In addition, any matching words in other caches are similarly updated.

Write Policy Noncachable memory: Only a portion of main memory is shared by more than one processor, and this is designated as noncachable. In such a system, all accesses to shared memory are cache misses, because the shared memory is never copied into the cache. The noncachable memory can be identified using chip-select logic or high-address bits.

Line size § As the block size increases from very small to larger sizes, the hit ratio will at first increase because of the principle of locality, which states that data in the vicinity(附近) of a referenced word is likely to be referenced in the near future. § As the block size increases, more useful data is brought into the cache. The hit ratio will begin to decrease, however, as the block becomes even bigger and the probability(概率) of using the newly fetched information becomes less than the probability of reusing the information that has to be replaced.

Line size • Larger blocks reduce the number of blocks that fit into a cache. Because each block fetch overwrites older cache contents, a small number of blocks results in data being overwritten shortly after it is fetched. • As a block becomes larger, each additional word is farther from the requested word, and therefore less likely to be needed in the near future. § The relationship between block size and hit ratio is complex, depending on the locality characteristics of a particular program, no definitive optimum value has been found.

Number of Cache § On-chip cache: when the requested instruction or data is found in the on-chip cache, the bus access is eliminated. Because of the short data paths internal to the processor, compared with bus lengths, on-chip cache accesses will complete appreciably faster than would even zero-wait(零等待) bus cycles. Furthermore, during this period the bus is free to support other transfers.

Number of Cache § External cache(L 2 cache) § Unified cache: Store both instructions and data • For a given cache size, higher hit rate than split caches • Only one cache needs to be designed and implemented § Split caches: One dedicated to instructions and one dedicated to data. • Evolving trend , particularly for superscalar machines • Eliminate contention for the cache between the instruction prefetcher(预取器) and the execution unit.