CPSC 614 Graduate Computer Architecture Memory Technology Based

CPSC 614: Graduate Computer Architecture Memory Technology Based on lectures by Prof. David Culler Prof. David Patterson UC Berkeley

Main Memory Background • Random Access Memory (vs. Serial Access Memory) • Different flavors at different levels – Physical Makeup (CMOS, DRAM) – Low Level Architectures (FPM, EDO, BEDO, SDRAM) • Cache uses SRAM: Static Random Access Memory – No refresh (6 transistors/bit vs. 1 transistor Size: DRAM/SRAM 4 -8, Cost/Cycle time: SRAM/DRAM 8 -16 • Main Memory is DRAM: Dynamic Random Access Memory – Dynamic since needs to be refreshed periodically (8 ms, 1% time) – Addresses divided into 2 halves (Memory as a 2 D matrix): » RAS or Row Access Strobe » CAS or Column Access Strobe

Static RAM (SRAM) • Six transistors in cross connected fashion – Provides regular AND inverted outputs – Implemented in CMOS process Single Port 6 -T SRAM Cell

SRAM Read Timing (typical) • t. AA (access time for address): how long it takes to get stable output after a change in address. • t. ACS (access time for chip select): how long it takes to get stable output after CS is asserted. • t. OE (output enable time): how long it takes for the three-state output buffers to leave the highimpedance state when OE and CS are both asserted. • t. OZ (output-disable time): how long it takes for the three-state output buffers to enter highimpedance state after OE or CS are negated. • t. OH (output-hold time): how long the output data remains valid after a change to the address inputs.

SRAM Read Timing (typical) stable ADDR stable t. AA Max(t. AA, t. ACS) CS_L t. OH t. ACS OE_L t. AA DOUT t. OZ t. OE t. OZ valid WE_L = HIGH t. OE valid

Dynamic RAM • SRAM cells exhibit high speed/poor density • DRAM: simple transistor/capacitor pairs in high density form Word Line C . . . Bit Line Sense Amp

Basic DRAM Cell • Planar Cell – Polysilicon-Diffusion Capacitance, Diffused Bitlines • Problem: Uses a lot of area (< 1 Mb) • You can’t just ride the process curve to shrink C (discussed later)

Advanced DRAM Cells • Stacked cell (Expand UP)

Advanced DRAM Cells • Trench Cell (Expand DOWN)

DRAM Operations • Write – Charge bitline HIGH or LOW and set wordline HIGH • Read – Bit line is precharged to a voltage halfway between HIGH and LOW, and then the word line is set HIGH. – Depending on the charge in the cap, the precharged bitline is pulled slightly higher or lower. – Sense Amp Detects change • Explains why Cap can’t shrink – Need to sufficiently drive bitline – Increase density => increase parasitic capacitance Word Line C . . . Bit Line Sense Amp

DRAM logical organization (4 Mbit) Column Decoder … Sense Amps & I/O 11 D A 0…A 10 Row Decoder … Q Memory Array (2, 048 x 2, 048) Storage Word Line Cell • Square root of bits per RAS/CAS

So, Why do I freaking care? • By it’s nature, DRAM isn’t built for speed – Reponse times dependent on capacitive circuit properties which get worse as density increases • DRAM process isn’t easy to integrate into CMOS process – DRAM is off chip – Connectors, wires, etc introduce slowness – IRAM efforts looking to integrating the two • Memory Architectures are designed to minimize impact of DRAM latency – Low Level: Memory chips – High Level memory designs. – You will pay $$$$$$ and then some $$$ for a good memory system.

So, Why do I freaking care? • 1960 -1985: Speed = ƒ(no. operations) • 1990 – Pipelined Execution & Fast Clock Rate – Out-of-Order execution – Superscalar Instruction Issue • 1998: Speed = ƒ(non-cached memory accesses) • What does this mean for – Compilers? , Operating Systems? , Algorithms? Data Structures?

4 Key DRAM Timing Parameters • t. RAC: minimum time from RAS line falling to the valid data output. – Quoted as the speed of a DRAM when buy – A typical 4 Mb DRAM t. RAC = 60 ns – Speed of DRAM since on purchase sheet? • t. RC: minimum time from the start of one row access to the start of the next. – t. RC = 110 ns for a 4 Mbit DRAM with a t. RAC of 60 ns • t. CAC: minimum time from CAS line falling to valid data output. – 15 ns for a 4 Mbit DRAM with a t. RAC of 60 ns • t. PC: minimum time from the start of one column access to the start of the next. – 35 ns for a 4 Mbit DRAM with a t. RAC of 60 ns

DRAM Read Timing • Every DRAM access begins at: RAS_L – The assertion of the RAS_L – 2 ways to read: early or late v. CAS A CAS_L WE_L 256 K x 8 DRAM 9 OE_L D 8 DRAM Read Cycle Time RAS_L CAS_L A Row Address Col Address Junk WE_L OE_L D High Z Junk Read Access Time Data Out Early Read Cycle: OE_L asserted before CAS_L High Z Output Enable Delay Data Out Late Read Cycle: OE_L asserted after CAS_L

DRAM Performance • A 60 ns (t. RAC) DRAM can – perform a row access only every 110 ns (t. RC) – perform column access (t. CAC) in 15 ns, but time between column accesses is at least 35 ns (t. PC). » In practice, external address delays and turning around buses make it 40 to 50 ns • These times do not include the time to drive the addresses off the microprocessor nor the memory controller overhead! • Can it be made faster?

Fast Page Mode DRAM • Page: All bits on the same ROW (Spatial Locality) – Don’t need to wait for wordline to recharge – Toggle CAS with new column address

Extended Data Out (EDO) • Overlap Data output w/ CAS toggle – Later brother: Burst EDO (CAS toggle used to get next addr)

Synchronous DRAM • Has a clock input. – Data output is in bursts w/ each element clocked • Flavors: SDRAM, DDR Write Read PC 100: Intel spec to meet 100 MHz memory bus designs. Introduced w/ i 440 BX chipset

RAMBUS (RDRAM) • Protocol based RAM w/ narrow (16 -bit) bus – High clock rate (400 Mhz), but long latency – Pipelined operation • Multiple arrays w/ data transferred on both edges of clock RAMBUS Bank RDRAM Memory System

RDRAM Timing

DRAM History • DRAMs: capacity +60%/yr, cost – 30%/yr – 2. 5 X cells/area, 1. 5 X die size in 3 years • ‘ 98 DRAM fab line costs $2 B – DRAM only: density, leakage v. speed • Rely on increasing no. of computers & memory per computer (60% market) – SIMM or DIMM is replaceable unit => computers use any generation DRAM • Commodity, second source industry => high volume, low profit, conservative – Little organization innovation in 20 years – Don’t want to be chip foundries (bad for RDRAM) • Order of importance: 1) Cost/bit 2) Capacity – First RAMBUS: 10 X BW, +30% cost => little impact

Main Memory Organizations • Simple: – CPU, Cache, Bus, Memory same width (32 or 64 bits) • Wide: – CPU/Mux 1 word; Mux/Cache, Bus, Memory N words (Alpha: 64 bits & 256 bits; Utra. SPARC 512) • Interleaved: – CPU, Cache, Bus 1 word: Memory N Modules (4 Modules); example is word interleaved

Main Memory Performance • Timing model (word size is 32 bits) – 1 to send address, – 6 access time, 1 to send data – Cache Block is 4 words • Simple M. P. = 4 x (1+6+1) = 32 • Wide M. P. = 1 + 6 + 1 = 8 • Interleaved M. P. = 1 + 6 + 4 x 1 = 11

Independent Memory Banks • Memory banks for independent accesses vs. faster sequential accesses – Multiprocessor – I/O – CPU with Hit under n Misses, Non-blocking Cache • Superbank: all memory active on one block transfer (or Bank) • Bank: portion within a superbank that is word interleaved (or Subbank) … Superbank Number Bank Superbank Offset Bank Number Bank Offset

Independent Memory Banks • How many banks? number banks number clocks to access word in bank – For sequential accesses, otherwise will return to original bank before it has next word ready • Increasing DRAM => fewer chips => less banks RIMM’s can have a HOTSPOT (literally)

Avoiding Bank Conflicts • Lots of banks int x[256][512]; for (j = 0; j < 512; j = j+1) for (i = 0; i < 256; i = i+1) x[i][j] = 2 * x[i][j]; • Even with 128 banks, since 512 is multiple of 128, conflict on word accesses • SW: loop interchange or declaring array not power of 2 (“array padding”) • HW: Prime number of banks – – – bank number = address mod number of banks address within bank = address / number of words in bank modulo & divide per memory access with prime no. banks? address within bank = address mod number words in bank number? easy if 2 N words per bank

Fast Bank Number • Chinese Remainder Theorem As long as two sets of integers ai and bi follow these rules and that ai and aj are co-prime. If i j, then the integer x has only one solution (unambiguous mapping): – bank number = b 0, number of banks = a 0 (= 3 in example) – address within bank = b 1, number of words in bank = a 1 (= 8 in example) – N word address 0 to N-1, prime no. banks, words power of 2 Bank Number: Address within Bank: 0 3 2 9 12 15 18 21 Seq. Interleaved 0 1 2 0 4 6 10 13 16 19 22 1 5 7 11 14 17 20 23 2 9 8 3 12 21 6 15 Modulo Interleaved 0 1 2 0 1 18 19 4 13 22 7 16 17 10 11 20 5 14 23 81 23 4 5 6 7

Minimum Memory Size DRAMs per PC over Time ‘ 86 1 Mb 32 4 MB 8 MB 16 MB 32 MB 64 MB DRAM Generation ‘ 89 ‘ 92 ‘ 96 ‘ 99 ‘ 02 4 Mb 16 Mb 64 Mb 256 Mb 1 Gb 8 16 4 8 2 4 1 8 2 128 MB 4 1 256 MB 8 2

Need for Error Correction! • Motivation: – Failures/time proportional to number of bits! – As DRAM cells shrink, more vulnerable • Went through period in which failure rate was low enough without error correction that people didn’t do correction – DRAM banks too large now – Servers always corrected memory systems • Basic idea: add redundancy through parity bits – Simple but wastful version: » Keep three copies of everything, vote to find right value » 200% overhead, so not good! – Common configuration: Random error correction » SEC-DED (single error correct, double error detect) » One example: 64 data bits + 8 parity bits (11% overhead) – Really want to handle failures of physical components as well » Organization is multiple DRAMs/SIMM, multiple SIMMs » Want to recover from failed DRAM and failed SIMM! » Requires more redundancy to do this » All major vendors thinking about this in high-end machines

Architecture in practice • (as reported in Microprocessor Report, Vol 13, No. 5) – Emotion Engine: 6. 2 GFLOPS, 75 million polygons per second – Graphics Synthesizer: 2. 4 Billion pixels per second – Claim: Toy Story realism brought to games!

FLASH Memory • Floating gate transitor – Presence of charge => “ 0” – Erase Electrically or UV (EPROM) • Peformance – Reads like DRAM (~ns) – Writes like DISK (~ms). Write is a complex operation

More esoteric Storage Technologies? • Tunneling Magnetic Junction RAM (TMJ-RAM): – Speed of SRAM, density of DRAM, non-volatile (no refresh) – New field called “Spintronics”: combination of quantum spin and electronics – Same technology used in high-density disk-drives • MEMs storage devices: – Large magnetic “sled” floating on top of lots of little read/write heads – Micromechanical actuators move the sled back and forth over the heads

Tunneling Magnetic Junction

MEMS-based Storage • Magnetic “sled” floats on array of read/write heads – Approx 250 Gbit/in 2 – Data rates: IBM: 250 MB/s w 1000 heads CMU: 3. 1 MB/s w 400 heads • Electrostatic actuators move media around to align it with heads – Sweep sled ± 50 m in < 0. 5 s • Capacity estimated to be in the 1 -10 GB in 10 cm 2 See Ganger et all: http: //www. lcs. ece. cmu. edu/research/MEMS

Main Memory Summary • Wider Memory • Interleaved Memory: for sequential or independent accesses • Avoiding bank conflicts: SW & HW • DRAM specific optimizations: page mode & Specialty DRAM • Need Error correction

Virtual Memory

Terminology • Page: a virtual memory block • Page fault: a virtual memory miss • Memory mapping (memory translation): converting a virtual memory produced by the CPU to a physical address

Mapping of Virtual Memory to Physical Memory

Typical Parameter Ranges Parameter First-level cache Virtual Memory Block size 16 ~ 128 B 4096~65, 536 B Hit time 1~3 cycles 50~150 cycles Miss penalty 8~150 cycles access time 6~130 cycles 1, 000 ~ 10, 000 cycles 800, 000 ~ 8, 000 cycles 200, 000 ~ 2, 000 cycles 0. 00001~0. 001% transfer time Miss rate 2~20 cycles 0. 1~10%

Design Issues • A page fault takes millions of cycles to process. – Pages should be large enough to amortize the high access time. (4 KB ~ 64 KB) – Fully associative placement of pages is used. – Page faults can be handled in software. – Write-back (Write-through scheme does not work. )

Where to Place a Page and How to Find it • Fully associative placement • A page table is used to located pages. – Resides in memory – Indexed with the page number from the virtual address and contains the corresponding physical page number. – Each program has its own page table. – To indicate the location of the page table in memory, the page table register is used. – A valid bit in each entry (off: the page is not in memory => page fault)

Translation of Virtual Address

Page Table Virtual page

Writes in Virtual Memory • Writes to the next level of memory hierarchy (disk) take millions of cycles. • Write-through (with write buffer) is not practical. • Write-back (copy back): Virtual memory systems perform the individual writes into the page in memory and copy the page back to disk when it is replaced. • Dirty bit: indicates the page has been modified.

TLB (Translation Lookaside Buffer) • Each memory access by a program takes at least twice as long. – One to obtain the physical address in the page table – One to get the data • TLB (Translation Lookaside Buffer) – A cache that holds only page table mapping – Includes the reference bit, the dirty bit, and the valid bit. – We don’t need to access the page table on every reference.

TLB Structure Dirty Valid Ref Virtual Page # Physical Page #

TLB Acting as a Cache on Page Table

TLB Design Issues • When a TLB entry is replaced, we need to copy the reference and dirty bits back to the page table entry. • Write-back (due to small miss rate) • Fully associative mapping (due to small TLB) – If larger TLBs are used, no or small associativity can be used. • Randomly choose an entry to replace.

Alpha 21264 Data TLB PID

MIPS R 2000 TLB Virtual address

Memory Hierarchy Processor First-level Cache TLB Second-level Cache address blocks Page Table Memory pages Disk