Optimization of LFS with Slack Space Recycling and

Optimization of LFS with Slack Space Recycling and Lazy Indirect Block Update Yongseok Oh <ysoh@uos. ac. kr> The 3 rd Annual Haifa Experimental Systems Conference May 24, 2010 Haifa, Israel Yongseok Oh and Donghee Lee (University of Seoul) Jongmoo Choi (Dankook University) Eunsam Kim and Sam H. Noh (Hongik University) SYSTOR 2010, Haifa Israel 1

Outline l Introduction l Slack Space Recycling and Lazy Indirect Block Update l Implementation of SSR-LFS l Performance Evaluation l Conclusions SYSTOR 2010, Haifa Israel 2

Log-structured File System A B C D Segment buffer Storage l LFS collects small write requests and writes them sequentially to the storage device [Rosenblum at el, ACM TOCS ‘ 91] l Advantages ■ Superior write performance for random workloads ■ Fast recovery l Drawbacks ■ On-demand cleaning ■ Cascading meta-data update SYSTOR 2010, Haifa Israel 3

Storage Devices l Hard Disk Drives ■ Mechanical components – disk head, spindle motor, and platter ■ Poor random read/write performance l Solid State Drives ■ Consist of NAND flash memory, no mechanical parts ■ High performance, low-power consumption, and shock resistance ■ Sequential writes are faster than random writes SYSTOR 2010, Haifa Israel 4

Outline l Introduction l Slack Space Recycling and Lazy Indirect Block Update l Implementation of SSR-LFS l Performance Evaluation l Conclusions SYSTOR 2010, Haifa Israel 5

LFS cleaning Copy A B Segment 1 C D Segment 2 A B C D Segment 3 l To make free segments ■ LFS cleaner copies valid blocks to other free segment l On-demand cleaning ■ Overall performance decreases l Background cleaning ■ It does not affect the performance SYSTOR 2010, Haifa Israel 6

Hole-plugging Copy A B Segment 1 C A B D Segment 2 Segment 3 l Matthews et al. employed Hole-plugging in LFS [Matthews et al, ACM OSR ’ 97] l The cleaner copies valid blocks to holes of other segments SYSTOR 2010, Haifa Israel 7

Slack Space Recycling (SSR) Scheme Segment buffer E F G H SSR A B E Segment 1 F C G H D Segment 2 Segment 3 l We proposed SSR scheme that directly recycles slack space to avoid ondemand cleaning l Slack Space is invalid area in used segment SYSTOR 2010, Haifa Israel 8

Cascading meta-data update Double Indirect update 1 C’ C’ A’ B Update D B’ A’ Data’ A C 1 N-1 N update B A Update C B A Data i-node B’ C D E N-1 N D’ F A’ Segment 1 B’ C’ D’ Segment 2 l Update of a data block incurs cascading meta-data update l Modification of a data block consumes 4 blocks SYSTOR 2010, Haifa Israel 9

Lazy Indirect Block Update (LIBU) scheme Indirect A Data Insert A Update Insert B B C C E 2 C’ E’ IBC (Indirect Block Cache) D E 1 N-1 N Update D B’ B’ C i-node Insert C’ A’ A Indirect Map B A’ Data’ Double Indirect F A’ Segment 1 1 2 N-1 N D’ D’ Segment 2 l We propose LIBU scheme to decrease cascading meta-data update ■ LIBU uses IBC (Indirect Block Cache) to absorb the frequent update of indirect blocks ■ Indirect Map is necessary to terminate cascading meta data update SYSTOR 2010, Haifa Israel 10

Crash Recovery i-node map Segment usage table indirect map flush Consistent state Rebuild Write Scan indirect blocks RAM Power failure Write Disk search the last checkpoint Check point(last) Check point l For crash recovery, LFS periodically stores checkpoint information l If power failure occurs, ■ search the last check point ■ scan all segments written after the last check point ■ rebuild i-node map, segment usage table, indirect map, and indirect blocks in IBC SYSTOR 2010, Haifa Israel 11

Outline l Introduction l Slack Space Recycling and Lazy Indirect Block Update l Implementation of SSR-LFS l Performance Evaluation l Conclusions SYSTOR 2010, Haifa Israel 12

Implementation of SSR-LFS l We implemented SSR-LFS (Slack Space Recycling LFS) ■ Using FUSE (Filesystem in Userspace) framework in Linux l SSR-LFS selectively chooses either SSR or cleaning ■ When the system is idle, it performs background cleaning ■ When the system is busy, it performs SSR or on-dmenad cleaning u If average slack size is too small, it selects on-demand cleaning u Otherwise, it selects SSR-LFS Core write(“/mnt/file”) IBC buffer cache i-node cache Syncer Cleaner Recycler libfuse Userspace Kernel VFS FUSE Architecture of SSR-LFS SYSTOR 2010, Haifa Israel 13

Outline l Introduction l Slack Space Recycling and Lazy Indirect Block Update l Implementation of SSR-LFS l Performance Evaluation l Conclusions SYSTOR 2010, Haifa Israel 14

Experimental Environment l For comparison, we used several file systems ■ Ext 3 -FUSE ■ Org-LFS(cleaning) ■ Org-LFS(plugging) ■ SSR-LFS l Benchmarks ■ IO TEST that generates random write workload ■ Postmark that simulates the workload of a mail server l Storage Devices ■ SSD – INTEL SSDSA 2 SH 032 G 1 GN ■ HDD – SEAGATE ST 3160815 AS SYSTOR 2010, Haifa Israel 15

Random Update Performance HDD SSD l SSR-LFS shows better than others for a wide range of utilization l On HDD ■ SSR-LFS and Org-LFS outperform Ext 3 -FUSE under utilization of 85%, l On SSD ■ Ext 3 -FUSE outperforms Org-LFS due to optimization of SSD for random writes SYSTOR 2010, Haifa Israel 16

Postmark Benchmark Result (1) SSD 6000 700 5000 600 Execution time HDD 4000 3000 2000 1000 500 400 300 200 100 0 0 Org-LFS SSR-LFS Ext 3 -FUSE l Medium file size (16 KB ~ 256 KB) ■ Subdirectories 1000, # of files 100, 000, # of transactions 100, 000 l SSR-LFS outperforms other file systems on both devices l Org-LFS shows better performance than Ext 3 -FUSE on HDD l Ext 3 -FUSE shows comparative performance to Org-LFS on SSD SYSTOR 2010, Haifa Israel 17

Postmark Benchmark Result (2) HDD SSD 7000 1200 1000 5000 Execution time 6000 4000 3000 2000 800 600 400 200 1000 0 Org-LFS SSR-LFS Ext 3 -FUSE l Small file size (4 KB ~ 16 KB) ■ Subdirectories 1000, # of files 500, 000 , # of transactions 200, 000 l Ext 3 -FUSE performs better than other file systems on SSD ■ due to meta-data optimization of Ext 3 such as hash-based directory SYSTOR 2010, Haifa Israel 18

Outline l Introduction l Slack Space Recycling and Lazy Indirect Block Update l Implementation of SSR-LFS l Performance Evaluation l Conclusions SYSTOR 2010, Haifa Israel 19

Conclusions l SSR-LFS outperforms original style LFS for a wide range of utilization l Future works ■ Optimization of meta-data structures ■ Cost-based selection between cleaning and SSR l We plan to release the source code of SSR-LFS this year ■ http: //embedded. uos. ac. kr SYSTOR 2010, Haifa Israel 20

Q&A l Thank you SYSTOR 2010, Haifa Israel 21

Back up slides SYSTOR 2010, Haifa Israel 22

Storage devices 200 150 100 Seq 50 0 Rand 250 200 150 100 Seq 50 0 Rand 4 K 8 K 16 K 32 K 64 K 128 K 256 K 512 K 1 M 2 M 4 M 8 M 16 M 250 Throughput(MB/s) Intel SSD (Read) 4 K 8 K 16 K 32 K 64 K 128 K 256 K 512 K 1 M 2 M 4 M 8 M 16 M Throughput(MB/s) Intel SSD (Write) Request Size 150 100 Seq 50 0 Rand Request Size 250 200 150 100 Seq 50 0 Rand 4 K 8 K 16 K 32 K 64 K 128 K 256 K 512 K 1 M 2 M 4 M 8 M 16 M 200 Throughput(MB/s) Seagate HDD (Read) 250 4 K 8 K 16 K 32 K 64 K 128 K 256 K 512 K 1 M 2 M 4 M 8 M 16 M Throughput(MB/s) Seagate HDD (Write) Request Size SYSTOR 2010, Haifa Israel 23

Measurement of FUSE overhead HDD SSD 80 250 60 50 40 Ext 3 30 Ext 3 -FUSE 20 Throughput(MB/s) 70 200 150 Ext 3 100 Ext 3 -FUSE 50 10 0 0 Seq-write rand-write seq-read rand-read l To identify the performance penalty of a user space implementation l Ext 3 -FUSE underperforms Kernel Ext 3 for almost patterns ■ Due to FUSE overhead SYSTOR 2010, Haifa Israel 24

Postmark Benchmark Result (1) SSD 6000 700 5000 600 Execution time HDD 4000 3000 2000 1000 500 400 300 200 100 0 0 Org-LFS 1302 cleanings SSR-LFS Ext 3 -FUSE Org-LFS SSR-LFS Ext 3 -FUSE 4142 SSRs l Medium file size (16 KB ~ 256 KB) ■ Subdirectories 1000, # of files 100, 000, # of transactions 100, 000 l SSR-LFS outperforms other file systems on both devices l Org-LFS shows better performance then Ext 3 -FUSE on HDD l Ext 3 -FUSE shows comparative performance to Org-LFS on SSD SYSTOR 2010, Haifa Israel 25

Postmark Benchmark Result (2) HDD SSD 7000 1200 1000 5000 Execution time 6000 4000 3000 2000 800 600 400 200 1000 0 Org-LFS 3018 cleanings SSR-LFS Ext 3 -FUSE 0 Org-LFS SSR-LFS Ext 3 -FUSE 1692 cleanings 1451 SSRs l Small file size (4 KB ~ 16 KB) ■ Subdirectories 1000, # of files 500, 000 , # of transactions 200, 000 l Ext 3 -FUSE performs better than other file systems on SSD ■ due to meta-data optimization of Ext 3 such as hash-based directory SYSTOR 2010, Haifa Israel 26

Postmark Benchmark Result (3) SSD HDD 500 1400 450 1200 350 Execution time 400 300 250 200 150 1000 800 600 400 200 50 0 0 Org-LFS SSR-LFS Ext 3 -FUSE l Large file size (disappeared in the paper) ■ File size 256 KB ~ 1 MB ■ Subdirectories 1000 ■ # of files 10, 000 ■ # of transactions 10, 000 SYSTOR 2010, Haifa Israel 27

Statistics of IO-TEST SSD (Utilizaton 85%) 18000 16000 14000 12000 10000 8000 6000 written 4000 read 2000 Mega bytes 18000 10000 8000 6000 read 2000 R -L FS lu (P le FS (C O rg -L FS -L rg O SYSTOR 2010, Haifa Israel SS ng ) gg i ng ) an i R -L FS SS (P rg -L FS O (C le lu an i gg in ng ) g) 0 FS -L rg written 4000 0 O Mega bytes HDD (Utilizaton 85%) 28

Random Update Performance l IO TEST benchmark ■ 1 st stage u Creates 64 KB files until 90% utilization u Randomly deletes files until target utilization ■ 2 nd stage u Randomly updates up to 4 GB of file capacity u Measures the elapsed time in this stage u LFS has no free segments u Including cleaning or SSR cost Create files Disk 90%(High threshold) update files 20%(desired. Random threshold) Delete files SYSTOR 2010, Haifa Israel 29

Experimental Environment Type Specifics OS Ubuntu 9. 10 (linux-2. 6. 31. 4) Processor AMD 2. 1 GHz Opteron 1352 RAM SAMSUNG 4 GB DDR-2 800 Mhz ECC SSD INTEL SSDSA 2 SH 032 G 1 GN HDD SEAGATE ST 3160815 AS SYSTOR 2010, Haifa Israel 30

Comparison of cleaning and SSR scheme Cleaning case : Write Request Tseg Foreground x delay Ton-demand Tseg cleaning write Tidle Tback-ground Background Free Segments 1 SSR case : 0 1 0 T 1 T 2 T 3 1 3 2 Time Write Request Foreground Tseg Tssr write SSR Tidle Tseg write Tback-ground Background Free Segments 1 0 T 1 0 1 2 T 2 SYSTOR 2010, Haifa Israel 3 Time 31

FIO Benchmark Result (1) HDD 140 80 120 70 100 80 Org-LFS 60 SSR-LFS Ext 3 -FUSE 40 20 Throughput(MB/s) SSD 60 50 Org-LFS 40 SSR-LFS 30 Ext 3 -FUSE 20 10 0 0 Seq-write rand-write seq-read rand-read l To measure the performance impact of Lazy Indirect Block Update l 1 worker - File size 1 GB, Request size 4 KB, Sync I/O mode l SSR-LFS outperforms Org-LFS and Ext 3 -FUSE for sequential and random writes on both devices SYSTOR 2010, Haifa Israel 32

FIO Benchmark Result (2) HDD 140 90 120 80 100 80 Org-LFS 60 SSR-LFS Ext 3 -FUSE 40 20 Throughtput(MB/s) Throughput(MB/s) SSD 70 60 50 Org-LFS 40 SSR-LFS 30 Ext 3 -FUSE 20 10 0 0 Seq-write rand-write seq-read rand-read l 4 workers - File size 1 GB, Request size 4 KB, Sync mode l The LIBU scheme has greater impact on performance when four workers are running SYSTOR 2010, Haifa Israel 33