Storage Systems ktwcsie ntu edu tw Embedded Systems

Storage Systems 郭大維 教授 ktw@csie. ntu. edu. tw 嵌入式系統暨無線網路實驗室 (Embedded Systems and Wireless Networking Laboratory) 國立臺灣大學資訊 程學系 Reading: Kam-yiu Lam and Tei-Wei Kuo, “Real-Time Database Systems: Architecture and Techniques”, Kluwer Academic Publishers, 2000 Krishna and Kang, “Real-Time. Systems, ” Mc. GRAW-HILL, 1997.

Storage Systems – Real-Time Disk Scheduling – Flash-Memory Storage Systems

Real-Time Disk Scheduling Motivation: Disparity between CPU and disk speed. access time = queuing time + seek time + latency delay + transfer time Strategies to improve the performance of disk service: First-come-first-served (FCFS) algorithm*: Poor because of no consideration in deadlines and arm movements. Earliest-deadline-first (EDF) algorithm: Not optimum in minimizing the number of transaction deadlines missed. 1 4 380 390 Track 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. Requests 1 2 3 4 Deadlines 20 40 60 80 Track 390 1 380 4 3

Real-Time Disk Scheduling Scan (or elevator) algorithm*: Start at one end of the disk, and moves toward the other end, servicing requests as it reaches each track, until it gets to the other end of the disk. At the other end, the direction of head movement is reversed and servicing continues. Bad for service requests at either end of a disk. 5 30 70 150 225 295 399 Track queue=295, 30, 150 C-Scan (Circle Scan) algorithm*: queue=5, 70, 225 Goal: Provide a more uniform wait time. As does Scan scheduling, servicing requests as it goes. However, when the head reaches one end, it immediately returns to the beginning of the disk. 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 4

Real-Time Disk Scheduling Shortest-seek-time-first (SSTF)algorithm*: A greedy algorithm which always selects the request with the minimum seek time from the current request queue. Starvation of some requests. . . 5 30 70 150 225 305 399 Track queue=305, 30, 5, 225 queue=305, 150 A variation of SCAN: 2021/3/11 Classify requests into classes. Service requests in the same class in terms of SCAN. Service classes in order of their priorities. Q: How many priority levels are enough, and how to partition them? * It means no consideration of deadlines. Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 5

Real-Time Disk Scheduling A weighted scheduling algorithm: Sort requests in the waiting queue in the increasing order of their deadlines. Each request is assigned a weight wi depending on their order in the queue. Let di be the distance the arm has to move from its current position to serve the request. Consider q requests at a time to reduce the algorithm complexity. Service the request with the highest priority pi = 1/ (wi di) Q: How to assign processes weights wi? A variation of the weighted scheduling algorithm: Motivation: Consider deadline instead of deadline order! Service the request with the highest priority pi = f(di , di) = a di + (1 -a) di. a is a design factor, and choosing a in the range 0. 7 to 0. 8 looks good. Reading: A. Silberschatz and P. B. Galvin, “Operating System Concepts, ” 4 th Ed. , Addison-Wesley Publishing Company, 1994. C. M. Krishna and K. G. Shin, “Real-Time. Systems, ” Mc. GRAW-HILL, 1997. S. Chen, J. A. Stankovic, J. F. Kurose, and D. F. Towsley, “Performance Evaluation of Two New disk scheduling Algorithms for Real-Time Systems, ” J. of Real-Time Systems, 3(3): 307 -336, 1991. 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 6

Real-Time Disk Scheduling Another paper for discussion: A. L. N. Reddy and J. C. Wyllie, “I/O Issues in Multimedia System, ” IEEE Transactions on Computers, March 1994. 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 7

Flash-Memory Storage Systems 郭大維 教授 ktw@csie. ntu. edu. tw 嵌入式系統暨無線網路實驗室 (Embedded Systems and Wireless Networking Laboratory) 國立臺灣大學資訊 程學系

Agenda Introduction Management Issues Performance vs Overheads Other Challenging Issues Conclusion 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 9

Introduction – Why Flash Memory Diversified Application Domains Portable Storage Devices Consumer Electronics Industrial Applications So. C and Hybrid Devices Critical System Components 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 10

Trends in VLSI Technology Source: www. icknowledge. com 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 11

11 March 2021 Trends in Flash Memory Development • Significant dropping of the price per GB – More than 40% per year • Exponential increasing in the density and capacity Source: IDC, Jun 2007 IBM Journal of Research and Development, Nov 2008 Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 12

11 March 2021 Trends in Flash Memory Development Market size: More than 15 billions US dollars in 2008 Revenue growth: More than 10% in 2008 IC Technology, NAND Flash Cost. Trend, 2008 13 Source: i. Suppli, Apr 2008 Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 13

Introduction – Trends in Storage Technology September 2006 Samsung 2 GB USB 2. 0 Flash Drive Price: $49. 99 Less Rebate: - $25. 00 Final Price: $24. 99* T-One 2 GB Microdrive/3600 RPM $144. 99 Source: Using multilevel cell NAND flash technology in consumer applications, Electronic Engineering Times, July , 2005 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 14

Introduction – Trends in Storage Technology March 2007 Transcend 8 GB Compact. Flash Card Price: $84. 85 Scan. Disk 4 GB Compact. Flash Card Price: $55. 99 Microdrive 4 GB Compact Flash Type II Source: Using multilevel cell NAND flash technology in consumer applications, Price: $116 Electronic Engineering Times, July , 2005. Amazon. com 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 15

Trends – Storage Media Seagate ST 302504 FDA 1 E 1 -RK Free. Agent Desktop 250 GB USB External Hard Drive (USD 79. 99) Transcend 8 GB SDHC SD CARD (USD 30. 30) March 2008 HITACHI 6 GB Microdrive MD 6 GBBP (USD 169. 95) Transcend 32 GB SSD, 2. 5 - Inch, SATA, MLC (USD 343. 30) Source: Using multilevel cell NAND flash technology in consumer applications, Electronic Engineering Times, July , 2005 Component Times, Nov 2007. * Parts of this slide was from the ASP-DAC’ 06 talk delivered by Prof. Sang. Lyul. Min from the Seoul National University. 2008/1/30 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 16

Trends – Storage Media 20 Gb IBM HDD Seagate Free. Agent 500 GB 3. 5 -Inch USB 2. 0 Hard Drive ST 305004 FDA 1 E 1 -RK (USD 85. 99) October 2008 Transcend 8 GB SDHC SD CARD (USD 19. 95) HITACHI 6 GB Microdrive MD 6 GBBP (USD 149. 99) Transcend 32 GB SSD, 2. 5 - Inch, SATA, MLC (USD 92. 24) Source: Using multilevel cell NAND flash technology in consumer applications, Electronic Engineering Times, July , 2005, and Component Times, Nov 2007. The first commercial flash chip was presented by Toshiba in 1985 as a 256 Kbit chip, where 1 Mb DRAM was introduced in 1986. 2008/11/11 * Parts of this slide was from the ASP-DAC’ 06 talk delivered by Prof. Sang. Lyul. Min from the Seoul National University. 17 Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. All Rights Preserved, Tei-Wei Kuo, National Taiwan University

Trends – Storage Media Seagate Free. Agent Desk 1 TB USB 2. 0 Hard Drive ST 310005 FDA 2 E 1 -RK (USD 119. 99) May 2009 Transcend 16 GB SDHC SD CARD (USD 39. 50) 4 GB HITACHI MICRODRIVE CF+ TYPE II TRAVEL KIT (MD 4 GB/ A HMS 360604 D 5 CF 00) (USD 25. 00) Transcend 2. 5" SSD 64 GB SATA-II (USD 144. 12) Source: Using multilevel cell NAND flash technology in consumer applications, Electronic Engineering Times, July , 2005, and Component Times, Nov 2007. The first commercial flash chip was presented by Toshiba in 1985 as a 256 Kbit chip, where 1 Mb DRAM was introduced in 1986. * Parts of this slide was from the ASP-DAC’ 06 talk delivered by Prof. Sang. Lyul. Min from the Seoul National University. 2021/3/11 18 Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. All Rights Preserved, Tei-Wei Kuo, National Taiwan University

Trends – Storage Media Seagate Free. Agent Desk 1. 5 TB USB 2. 0 Hard Drive ST 315005 FDA 2 E 1 -RK (USD 145. 77) November 2009 Transcend 16 GB SDHC SD CARD (USD 36. 49) 4 GB HITACHI MICRODRIVE CF (USD 19. 00) Transcend 128 GB SATA-2 2. 5 -inch Solid State (USD 283. 99) Source: Using multilevel cell NAND flash technology in consumer applications, Electronic Engineering Times, July , 2005, and Component Times, Nov 2007. The first commercial flash chip was presented by Toshiba in 1985 as a 256 Kbit chip, where 1 Mb DRAM was introduced in 1986. * Parts of this slide was from the ASP-DAC’ 06 talk delivered by Prof. Sang. Lyul. Min from the Seoul National University. 2021/3/11 19 Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. All Rights Preserved, Tei-Wei Kuo, National Taiwan University

Trends – Storage Media Western Digital WD Elements 2 TB USB 2. 0 Desktop External Hard Drive WDBAAU 0020 HBK-NESN (USD 109. 99) November 2010 Transcend 32 GB SDHC SD CARD (USD 50. 99) Kingston SSDNow V Series 128 GB SATA 2. 5 - Inch Solid State Drive (USD 224. 35) Source: Using multilevel cell NAND flash technology in consumer applications, Electronic Engineering Times, July , 2005, and Component Times, Nov 2007. The first commercial flash chip was presented by Toshiba in 1985 as a 256 Kbit chip, where 1 Mb DRAM was introduced in 1986. * Parts of this slide was from the ASP-DAC’ 06 talk delivered by Prof. Sang. Lyul. Min from the Seoul National University. 2021/3/11 20 Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. All Rights Preserved, Tei-Wei Kuo, National Taiwan University

The Ultimate Limit on Mechanical Devices u. A Microdrive Example Fly By Night 2, 000 Miles Per Hour Boeing 747 1/100” Flying Height Source: Richard Lary, The New Storage Landscape: Forces shaping the storage economy, 2003. 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. Source: http: //www. hitachigst. com/ 21

Flash Makers – NAND Flash Memory [EE Times, 11/30/2007] For years, NAND prices have dropped by an average of 40 percent or more per year. Source: i. Suppli Corp (Unit: Million Dollars) http: //hugoleijtens. spaces. live. com/blog/cns!4 B 94 B 7453 D 4 BFD 9 E!988. entry Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 22

Flash Stories – Solid-State Disks < 50% Heat, Ultra Silence, Light Weight, MTTF – 200 M hours (HDD – 30 M hours), Energy Efficiency Embedded Systems and Wireless Networking Lab. IDC, San. Disk - Component Times, Nov 2007 Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 23

Flash Stories – Flash Wars Fab 300 mm Wafer Fab by Toshiba and San. Disk at Yokkaichi, Japan 80, 000 Wafers per Month (2008) 210, 000 Wafers per Month 300 mm Wafer Fab by IM Flash Technologies (Intel & Micron) at Utah Joint Venture: Sony and Qimonda, Hynix and Sandisk Technology 32 nm (Samsung, Intel), 43 nm (Toshiba), 50 nm (IM) 2008/1/30 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 24

Introduction – The Characteristics of Storage Media Access time Media DRAM NOR FLASH 13 X NAND FLASH DISK Read Write Erase 5 ns (1 B) 2. 56 us (512 B) - 14 us (1 B) 7. 2 ms (512 B) 18 ms (128 KB) 200 us (1 B) 212 us (512 B) 1. 5 ms (128 KB) 45 ns (1 B) 83 X 23 us (512 B) 20 us (1 B) 32. 8 us (512 B) 390 X 12. 7 ms (512 B) 67 X 14. 2 ms(512 B) (average) - (average) Reference Devices/Modules: DRAM: DDR-400. NOR FLASH: Silicon Storage Technology SST 39 LF 010. NAND FLASH: Samsung K 9 K 8 G 08 U 0 M. Disk: Seagate ST 31000333 AS

Non-Volatile Memory Macronix International Co. , Component Times, Nov. 2007 2008/1/30 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 26

Introduction – Single-Level Cell (SLC) Each Word Line is connected to control gates. Each Bit Line is connected to the drain. Control Gate Drain Source Cell NOR Flash IDS Selected cell All Rights Preserved, Tei-Wei Kuo, National Taiwan University Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 27

Introduction – Single-Level Cell (SLC) NAND accesses each cell through adjacent cells, while NOR allows for individual access to each cell. 1 The cell size of NAND is almost half the size of a NOR cell. "Flash Memory Moves from Niche to Mainstream, " Chip Design Magazine, April/May 2006. * NOR Flash All Rights Preserved, Tei-Wei Kuo, National Taiwan University Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. NAND Flash 28

Introduction – Multi-Level Cell (MLC) vs SLC 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 29

Comparison of SLC and MLC 1 -bit/Cell SLC NAND Flash 100, 000 Program/Erase cycles (with ECC)[1] 10 years Data Retention[1] 2 -bits/Cell MLC NAND Flash 10, 000 Program/Erase cycles (with ECC) [2] 10 years Data Retention[2] 4 -bits/Cell MLC NAND FLASH Developers (2006) M-systems, Intel, Samsung, and Toshiba [1] ST Micro-electronics NAND SLC large page datasheet (NAND 08 GW 3 B 2 A) [2] ST Micro-electronics NAND MLC large page datasheet (NAND 04 GW 3 C 2 A) * USD 34. 65 per GB for NOR, USD 6. 79 per GB for NAND (March 2007) 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 30

Introduction – Consumer Applications 2021/3/11 Electronic Engineering Times, Embedded Systems and Wireless Networking Lab. July 2005 Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 31

Bandwidth Requirements – Video ˇ ˇ Electronic Engineering Times, July 2005 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 32

Bandwidth Requirements – Audio ˇ ˇ Electronic Engineering Times, July 2005 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 33

Introduction – Challenges in Flash. Memory Storage Designs Requirements in Good Performance Limited Cost per Unit Strong Demands in Reliability Increasing in Access Frequencies Tight Coupling with Other Components Low Compatibility among Vendors 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 34

Agenda Introduction Management Issues Performance vs Overheads Other Challenging Issues Conclusion 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 35

Management Issues – System Architectures AP AP File-System Layer Block Device Layer (FTL emulation) MTD drivers Flash Memory 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 36

Management Issues – Flash-Memory Characteristics 1 Page = 512 B 1 Block = 32 pages(16 KB) Write one page Block 0 Block 1 Block 2 Block 3 …… …… 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. Erase one block 37

Management Issues – Flash-Memory Characteristics Write-Once No writing on the same page unless its residing block is erased! Pages are classified into valid, invalid, and free pages. Bulk-Erasing Pages are erased in a block unit to recycle used but invalid pages. Wear-Leveling Each block has a limited lifetime in erasing counts. 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 38

Management Issues – Flash-Memory Characteristics Example 1: Out-place Update A B C D Live pages Free pages Suppose that we want to update data A and B… 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 39

Management Issues – Flash-Memory Characteristics Example 1: Out-place Update A B C D A B Dead pages 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 40

Management Issues – Flash-Memory Characteristics Example 2: Garbage Collection L D D L D L L L F D This block is to be recycled. (3 live pages and 5 dead pages) L F L L D F F L L F D 2021/3/11 A live page A dead page A free page Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 41

Management Issues – Flash-Memory Characteristics Example 2: Garbage Collection D D D D Live data are copied to somewhere else. L L D L F L L D L L F D 2021/3/11 A live page A dead page A free page Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 42

Management Issues – Flash-Memory Characteristics Example 2: Garbage Collection F F F F The block is then erased. L L D Overheads: • live data copying • block erasing. L F L L D L L F D 2021/3/11 A live page A dead page A free page Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 43

Management Issues – Flash-Memory Characteristics Example 3: Wear-Leveling 100 L D D L D A 10 L L D L L L F D B 20 L F L L D F C 15 F L L F D D Wear-leveling might interfere with the decisions of the blockrecycling policy. A live page A dead page A free page Erase cycle counts 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 44

Management Issues – Challenges The write throughput drops significantly after garbage collection starts! The capacity of flash-memory storage systems increases very quickly such that memory space requirements grows quickly. Reliability becomes more and more critical when the manufacturing capacity increases! The significant increment of flash-memory access rates seriously exaggerates the Read/Program Disturb Problems! 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 45

Agenda Introduction Management Issues Performance vs Overheads – FTL vs NFTL Other Challenging Issues Conclusion 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 46

System Architecture fwrite(file, data) process File system (FAT, EXT 2, NTFS. . . ) Applications File Systems Block write (LBA, size) Flash I/O Requests Garbage Collection Address Translation FTL/NFTL Layer Flash-Memory Storage System Device Driver Control signals 2021/3/11 Physical Devices (Flash Memory Banks) Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 47

Management Issues – Flash. Memory Characteristics SD, Memory Stick, Compact Flash x. D, Smart. Media *FTL: Flash Translation Layer, MTD: Memory Technology Device

Policies – FTL adopts a page-level address translation mechanism. The main problem of FTL is on large memory space requirements for storing the address translation information. 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 49

Policies – NFTL (Type 1) A logical address under NFTL is divided into a virtual block address and a block offset. e. g. , LBA=1011 => virtual block address (VBA) = 1011 / 8 = 126 and block offset = 1011 % 8 = 3 NFTL Address Translation Table Write data to (in main-memory) A Chain Block Address = 9 Address = 23 Address = 50 Free Free Free Used Free Free LBA=1011 . . . (9) VBA=126 2021/3/11 . . . Block Offset=3 Write. Iftothe thepage. Write has to the Freewith block Free page with block page been used offset=3 Free Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. Free 50

Policies – NFTL (Type 2) A logical address under NFTL is divided into a virtual block address and a block offset. e. g. , LBA=1011 => virtual block address (VBA) = 1011 / 8 = 126 and block offset = 1011 % 8 = 3 NFTL A Primary Block Address Translation Table (in main-memory) Address = 9 Write data to LBA=1011 . . . (9, 23) VBA=126 2021/3/11 . . . Block Offset=3 Embedded Systems and Wireless Networking Lab. A Replacement Block Address = 23 Free Used Free If the page has been Write to theused Free first free page Free Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. Free 51

Policies – NFTL is proposed for the large-scale NAND flash storage systems because NFTL adopts a block-level address translation. However, the address translation performance of read and write requests might deteriorate, due to linear searches of address translation information in primary and replacement blocks. 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 52

Policies – BL BL adopts a block-level address translation mechanism. The main problem of BL is on low space utilization and significant write overhead e. g. , LBA=1011 => virtual block address (VBA) = 1011 / 8 = 126 and block offset = 1011 % 8 = 3 Address Translation Table (in main-memory) Write data to LBA=1011 . . . 10 9 VBA=12 6 . . . Write Block Offset=3 Physical block = 9 Used Free Used Free Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. Copy Write Physical block = 10 Free Free 53

Policies – FTL or NFTL Memory Space Requirements Address Translation Time Garbage Collection Overhead Space Utilization FTL Large Short Less High NFTL Small Long More Low The Memory Space Requirements for one 1 GB NAND (512 B/Page, 4 B/Table Entry, 32 Pages/Block) FTL: 8, 192 KB (= 4*(1024*1024)/512) NFTL: 256 KB (= 4*(1024*1024)/(512*32)) Remark: Each page of small-block(/large-block) SLC NAND can store 512 B(/2 KB) data, and there are 32(/64) pages per block. Each page of MLCx 2 NAND can store 2 KB, and there are 128 pages per block. 2021/3/11 Embedded Systems and Wireless Networking Lab. 54 Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University.

Address Translation Time - NFTL The address translation performance of read and write requests can be deteriorated, due to linear searches of physical addresses. 1. Assume that each block contains 8 pages. 2. Let LBA A, B, C, D, and E be written for 5, 5, 1, 1, and 1 times, respectively. Their data distribution could be like to what in the left figure. 3. For example, it might need to scan 9 spare areas for LBA B. 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 55

Garbage Collection Overhead - NFTL 1. Copy the most-recent content to the new primary block. 2. Erase the old primary block and the replacement block. 3. Overhead is 2 block erases and 5 page writes. 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 56

Space Utilization - NFTL 3 free pages are wasted. 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 57

Agenda Introduction Management Issues Performance vs Overheads – An Adaptive Two-Level Mapping Mechanism Other Challenging Issues Conclusion 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 58

Motivation An adaptive two-level management design of a flash translation layer, called AFTL. Exploit the advantages of the fine-grained address mechanism and the coarse-grained address mechanism. FTL NFTL AFTL Large Small A little larger than NFTL Address Translation Short Time Long Much Better than NFTL Garbage Collection Overhead Less More Much Better than NFTL Space Utilization High Low Much Better than NFTL Memory Space Requirements 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 59

AFTL – Coarse-to-Fine Switching 1. AFTL doesn’t erase the two blocks immediately. 2. AFTL moves the mapping information of the replacement block to the fine-grained hash table by adding finegrained slots. 3. The RPBA field of the corresponding mapping information is nullified. 2021/3/11 Chin-Hsien Wu and Tei-Wei Kuo, 2006, “An Adaptive Two-Level Management for the Flash Translation Layer in Embedded Systems, ” IEEE/ACM 2006 International Conference on Computer-Aided Design (ICCAD), November 5 -9, 2006. Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 60

AFTL – Fine-to-Coarse Switching The number of the fine-grained slots is limited. Some least recently used mapping information of fine-grained slots should be moved to the coarse-grained hash table. (F, PBAE) 2021/3/11 1. Assume that this fine-grained slot is to be replaced. 2. Data stored in the page with the given (physical) address are copied to the primary or replacement block of the corresponding coarse-grained slot, as defined by NFTL. 3. If there dose not exist any corresponding coarse-grained slot, a new one is created. Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 61

AFTL – Fine-to-Coarse Switching Coarse-to-fine switches would introduce fine-tocoarse switches and overhead in valid page copying. It is because the number of the fine-grained slots is limited. Stop any coarse-to-fine switch when some frequency bound in coarse-to-fine switches is reached. We set a parameter in the experiments to control the frequency of switches to explore the behavior of the proposed mechanism. 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 62

The Advantages of AFTL Improve the address translation performance. It is because the moving of their mapping information to the fine-grained hash table. Improve the garbage collection overhead. The delayed recycling of any replacement block reduces the potential number of valid data copyings and blocks erased. Improve the space utilization. The delayed recycling of any primary block lets free pages of a primary block be likely used in the future. Chin-Hsien Wu and Tei-Wei Kuo, 2006, “An Adaptive Two-Level Management for the Flash Translation Layer in Embedded Systems, ” IEEE/ACM 2006 International Conference on Computer-Aided Design (ICCAD), November 5 -9, 2006. 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 63

Agenda Introduction Management Issues Performance vs Overheads – Performance Evaluation Other Challenging Issues Conclusion 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 64

Performance Evaluation Performance Setup The characteristics of the experiment trace was over a 20 GB disk. CPU Intel Celeron 750 MHz RAM 320 MB OS Windows XP File Systems NTFS Applications Web Applications, E-mail Clients, MP 3 Player, MSN Messenger, Word, Excel, Power. Point, Media, Player, Programming, and Virtual Memory Activities Durations One week Total Write / Read Requests 13, 198, 805 / 2, 797, 996 sectors Different LBA’s 1, 669, 228 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 65

Performance Evaluation Performance Setup The maximum number of fine-grained slots is controlled by a parameter MFS. A parameter ST controls the frequency of switches between the two address translation mechanisms – n/ST. ST=0 => No constraint on the number of switches. Smaller ST => More switches. Larger ST => Less switches. 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 66

Memory Space Requirements 1. MFS ranged from 2, 500, 5, 000, 7, 500, 10, 000, 12, 500, to 15, 000. 2. AFTL uses a little more memory space than NFTL. 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 67

Address Translation Performance 1. Larger MFS => smaller address translation time - More address translations going through the fine-grained address translation mechanism. 2. Smaller ST => longer address translation time - More coarse-to-fine switches 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 68

Garbage Collection Overhead AFTL outperforms NFTL. - Coarse-to-fine switches can avoid immediate recycling of their primary and replacement blocks and related valid data copyings. 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 69

Space Utilization The Space utilization might be better under AFTL. - Coarse-to-fine switches can delay the recycling of replacement blocks. - Free pages of primary blocks might be used in the future. . 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 70

Summary AFTL is proposed to exploit the advantages of finegrained/coarse-grained address translation mechanisms, and to switch dynamically and adaptively the mapping information between the two address translation mechanisms. AFTL does provide good performance in address mapping and space utilization and have garbage collection overhead and memory space requirements under proper management. 2021/3/11 Chin-Hsien Wu and Tei-Wei Kuo, 2006, “An Adaptive Two-Level Management for the Flash Translation Layer in Embedded Systems, ” Embedded Systems and Wireless Networking Lab. 71 IEEE/ACM 2006 International Conference on Computer-Aided Design (ICCAD), November 5 -9, 2006. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University.

Agenda Introduction Management Issues Performance vs Overheads Other Challenging Issues Conclusion 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 72

Challenging Issues – Reliability Each Word Line is connected to control gates. Each Bit Line is connected to the drain. Control Gate Drain Source IDS 2021/3/11 Cell Selected cell Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 73

Challenging Issues – Reliability Program Operation Read Operation When the floating gate is not charged with electrons, there is current ID (100 u. A) if a reading voltage is applied. (“ 1” state) Electrons are moved into the floating gate, and the threshold voltage is thus raised. 5 V 1 V 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 74

Challenging Issues – Reliability Over-Erasing Problems Fast Erasing Bits All of the cells connected to the same bit line of a depleted cell would be read as “ 1”, regardless of their values. Read/Program Disturb Problems DC erasing of a programmed cell, DC programming of a non-programmed cell, drain disturb, etc. Flash memory that has thin gate oxide makes disturb problems more serious! Data Retention Problems Electrons stored in a floating gate might be lost such that the lost of electrons will sooner or later affects the charging status of the gate! 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 75

Program Disturb Example - DC Erasing of a Programmed Cell Electrons might be tunneled from floating gate to control gate through interpoly oxide in all the programmed cells. E. g. , 12 V applied to the control gate and 0 V applied to both source and drain 2021/3/11 Cell A was programmed. Embedded Systems and Wireless Networking Lab. Programming Cell B also erases Cell A at the same time. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 76

Program Disturb Example – Drain Disturb Cells sharing the BL are to be programmed, since electrons are tunneled from the floating gate through gate oxide to the drain. Cell D was programmed. Programming Cell B also erases Cell D at the same time. 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 77

Challenging Issues – Observations The write throughput drops significantly after garbage collection starts! The capacity of flash-memory storage systems increases very quickly such that memory space requirements grows quickly. Reliability becomes more and more critical when the manufacturing capacity increases! The significant increment of flash-memory access rates seriously exaggerates the Read/Program Disturb Problems! Wear-leveling technology is even more critical when flash memory is adopted in many system components or might survive in products for a long life time! 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 78

Wear Leveling versus Product Lifetime Settings File system: FAT 16 file system with 8 KB cluster size Flash memory: 256 MB small-block flash memory with 100 K erase cycles Updating of a 16 MB file repeatedly with the throughput: 0. 1 MBs The file requires 2 K clusters = 16 MB ÷ 8 KB(cluster size) The FAT size of this file is 4 KB (2 K(clusters) x 2 bytes) The 20% of blocks in flash memory joins the dynamic wear leveling Data of a 16 MB file is stored in 1 K blocks (16 MB ÷ 16 KB(block size)) Suppose flash memory is managed in the block level File systems update the FAT in each cluster writing so that FAT is updated 2 K times for a 16 MB file Writing of a 16 MB incurs 1 K block erases because of the reclaiming of invalid space. 2008/1/30 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 79

Wear Leveling versus Product Lifetime Ways in Data Updates In-Place-Updates: Rewriting on the Same Page Dynamic Wear Leveling: Rewriting over Another Free Page with Erasing over Blocks with Dead Pages Static Wear Leveling: Rewriting over Another Free Page with Erasing over Any Blocks Expected Lifetime of 2008/1/30 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 80

Comparison of Different Technologies No Wear Leveling Dynamic Wear Leveling 5000 4000 3000 2000 1000 0 0 Erase Cycles 5000 20 0 40 60 80 100 0 20 Erase Cycles 5000 4000 3000 2000 1000 81 0 20 40 60 80 100 Physical Block Addresses (PBA) 60 80 100 Intuitive SWL Perfect SWL 0 40 0 0 20 40 60 Physical Block Addresses (PBA) Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University.

Wear Leveling versus Product Lifetime Static Wear Leveling – Block-Level Mapping Use a counter for each block The garbage collector always finds the block with the least erase count. Some heuristic approach erases a block to maintain 2 free blocks when the garbage collector finds the erase count of the block is over a given threshold. Problems: High extra block erases and live-page copyings High main-memory consumption High computation cost GC starts find a victim GC starts block Update Write new data in 15 block to block 3 4 block 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 : index in the selection of a victim block : index to the selected free block 5 54 5 5 54 4 5 5 5 4 4 5 Flash Memory Counter : block contains (some) valid data : free block 2008/1/30 : dead block Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 82

An Efficient Static Wear Leveling Mechanism A modular design for compatibility considerations A SWL mechanism Block Erasing Table (BET) bit flags SW Leveler SWL-Procedure SWL-Update 83 Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University.

The Block Erasing Table (BET) A bit-array: Each bit is for 2 k consecutive blocks. Small k – in favor of hot-cold data separation Large k – in favor of small RAM space ecnt=3 =2 =1 =0 fcnt =2 =0 =1 Flash 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 BET 0 0 10 0 0 01 10 0 k=2 ecnt: the total number of block erases done since the BET is reset fcnt: the number of 1’s in the BET 84 : an index to a block that the Cleaner wants to erase : a block that has been erased in the current resetting interval Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. ecnt=1 =2 =3 =4 =0 fcnt =1 =0 =2

The SW Leveler An unevenness level (ecnt / fcnt) >= T Triggering of the SW Leveler Resetting of BET when all flags are set. 0 1 2 3 4 5 6 7 8 ecnt =1999 =2000 =2004 =2999 =3000 =3004 =4000 =1998 =2998 =3999 =0 fcnt =0 =2 =3 =4 9 10 11 12 13 14 15 T: A threshold, T=1000 in this example ecnt: the total number of block erases since the BET is erased fcnt: The number of 1’s in the BET 10 01 k=2 : An index that SW Leveler triggers the Cleaner to do garbage collection : An index in the selection of a block set : An index to a block that the Cleaner wants to erase : A block that has been erased in the current resetting interval The Cleaner is triggered to Reset to a randomly 1. Copy valid data of selected block set to free area, After a period of time, the total erase count 4000 / 4 = 1000>=1000 (e selected block set cnt/ fcnt>=1000) 2. Erase block in the selected block set, and (flag) , but all flags in BET are 1 reset BET reaches 3999. count reaches 2998. 3. Inform the Allocator to update the address mapping 2000 / 2 = 1000 >= 1000 (E 3000 / 3 = 1000 >= 1000 (E cnt / fcnt >= T) between LBA and PBA 85 Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University.

Main-Memory Requirements 512 MB k=0 k=1 k=3 256 B 128 B 64 B 32 B 1 GB 512 B 256 B 128 B 64 B 2 GB 4 GB 8 GB 1024 B 512 B 256 B 128 B 2048 B 1024 B 512 B 256 B 4096 B 2048 B 1024 B 512 B MLCx 2 (1 page = 2 KB, 1 block=128 pages) 86 Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University.

Performance Evaluation Metrics Endurance Enhancement (First Failure Time) Additional Overheads (Extra Block Erases) Experiment Setup The FTL Layer FTL (Flash Translation Layer protocol) NFTL (NAND Flash Translation Layer protocol) Traces: A mobile PC with a 20 GB hard disk Duration: One month. Daily activities The average number of Operations: 1. 82 writes/sec, 1. 97 reads/sec The percentage of accessed LBA’s: 36. 62% 87 Flash Memory: 1 GB MLCx 2 Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University.

Endurance Improvement - The First Failure Time When k=3 and T=100, the endurance is improved by 100. 2%. When k=0 and T=100, the endurance is improved by 87. 5%. 88 Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University.

Additional Overhead - Extra Block Erases Extra block erases is less than 3%. 89 Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. Extra block erases is less than 2%.

Conclusion Summary The Characteristics of Flash Memory and Management Issues Popular Implementations: FTL vs NFTL Adaptive Two-Level Address Translation Performance, Cost, and Reliability Challenges 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 90

Conclusion What Is Happening? Solid-State Storage Devices New Designs in the Memory Hierarchy More Applications in System Components and Products Challenging Issues: Performance, Cost, and Reliability Scalability Technology Reliability Technology Customization Technology 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 91

Contact Information Professor Tei-Wei Kuo ktw@csie. ntu. edu. tw l URL: http: //csie. ntu. edu. tw/~ktw l Flash Research: http: //newslab. csie. ntu. edu. tw/~flash/ l Office: +886 -2 -23625336 -257 l Fax: +886 -2 -23628167 l Address: Dept. of Computer Science & Information Engr. National Taiwan University, Taipei, Taiwan 106 l 2021/3/11 Embedded Systems and Wireless Networking Lab. Copyright: All rights reserved, Prof. Tei-Wei Kuo, Embedded System and Wireless Networking Lab, National Taiwan University. 92

Q & A