Chapter 12 File System Implementation Operating System Concepts

Chapter 12: File System Implementation Operating System Concepts – 9 th Edition Silberschatz, Galvin and Gagne © 2013

Chapter 12: File System Implementation n File-System Structure n File-System Implementation n Directory Implementation n Allocation Methods n Free-Space Management n Efficiency and Performance n Recovery n NFS n Example: WAFL File System Operating System Concepts – 9 th Edition 12. 2 Silberschatz, Galvin and Gagne © 2013

Objectives n To describe the details of implementing local file systems and directory structures n To describe the implementation of remote file systems n To discuss block allocation and free-block algorithms and trade-offs Operating System Concepts – 9 th Edition 12. 3 Silberschatz, Galvin and Gagne © 2013

File-System Structure n File structure l Logical storage unit l Collection of related information n File system resides on secondary storage (disks) l Provided user interface to storage, mapping logical to physical l Provides efficient and convenient access to disk by allowing data to be stored, located retrieved easily n Disk provides in-place rewrite and random access l I/O transfers performed in blocks of sectors (usually 512 bytes) n File control block – storage structure consisting of information about a file n Device driver controls the physical device Operating System Concepts – 9 th Edition 12. 4 Silberschatz, Galvin and Gagne © 2013

File Systems n Many file systems, sometimes many within an operating system l Each with its own format (CD-ROM is ISO 9660; Unix has UFS, FFS; Windows has FAT, FAT 32, NTFS as well as floppy, CD, DVD Blu-ray, Linux has more than 40 types, with extended file system ext 2 and ext 3 leading; plus distributed file systems, etc. ) l New ones still arriving – ZFS, Google. FS, Oracle ASM, FUSE Operating System Concepts – 9 th Edition 12. 5 Silberschatz, Galvin and Gagne © 2013

File-System Implementation n We have system calls at the API level, but how do we implement their functions? l On-disk and in-memory structures n Boot control block contains info needed by system to boot OS from that volume l Needed if volume contains OS, usually first block of volume n Volume control block (superblock, master file table) contains volume details l Total # of blocks, # of free blocks, block size, free block pointers or array n Directory structure organizes the files l Names and inode numbers, master file table Operating System Concepts – 9 th Edition 12. 6 Silberschatz, Galvin and Gagne © 2013

File-System Implementation (Cont. ) n Per-file File Control Block (FCB) contains many details about the file l inode number, permissions, size, dates l NFTS stores into in master file table using relational DB structures Operating System Concepts – 9 th Edition 12. 7 Silberschatz, Galvin and Gagne © 2013

In-Memory File System Structures n Mount table storing file system mounts, mount points, file system types n The following figure illustrates the necessary file system structures provided by the operating systems n Figure 12 -3(a) refers to opening a file n Figure 12 -3(b) refers to reading a file n Plus buffers hold data blocks from secondary storage n Open returns a file handle for subsequent use n Data from read eventually copied to specified user process memory address Operating System Concepts – 9 th Edition 12. 8 Silberschatz, Galvin and Gagne © 2013

In-Memory File System Structures Operating System Concepts – 9 th Edition 12. 9 Silberschatz, Galvin and Gagne © 2013

Virtual File Systems n Virtual File Systems (VFS) on Unix provide an object-oriented way of implementing file systems n VFS allows the same system call interface (the API) to be used for different types of file systems l Separates file-system generic operations from implementation details l Implementation can be one of many file systems types, or network file system 4 Implements vnodes which hold inodes or network file details l Then dispatches operation to appropriate file system implementation routines Operating System Concepts – 9 th Edition 12. 10 Silberschatz, Galvin and Gagne © 2013

Virtual File Systems (Cont. ) n The API is to the VFS interface, rather than any specific type of file system Operating System Concepts – 9 th Edition 12. 11 Silberschatz, Galvin and Gagne © 2013

Virtual File System Implementation n For example, Linux has four object types: l inode, file, superblock, dentry n VFS defines set of operations on the objects that must be implemented l Every object has a pointer to a function table 4 Function table has addresses of routines to implement that function on that object 4 For example: 4 • int open(. . . )—Open a file 4 • int close(. . . )—Close an already-open file 4 • ssize t read(. . . )—Read from a file 4 • ssize t write(. . . )—Write to a file 4 • int mmap(. . . )—Memory-map a file Operating System Concepts – 9 th Edition 12. 12 Silberschatz, Galvin and Gagne © 2013

Directory Implementation n Linear list of file names with pointer to the data blocks l Simple to program l Time-consuming to execute 4 Linear search time 4 Could keep ordered alphabetically via linked list or use B+ tree n Hash Table – linear list with hash data structure l Decreases directory search time l Collisions – situations where two file names hash to the same location l Only good if entries are fixed size, or use chained-overflow method Operating System Concepts – 9 th Edition 12. 13 Silberschatz, Galvin and Gagne © 2013

Allocation Methods - Contiguous n An allocation method refers to how disk blocks are allocated for files: n Contiguous allocation – each file occupies set of contiguous blocks l Best performance in most cases l Simple – only starting location (block #) and length (number of blocks) are required l Problems include finding space for file, knowing file size, external fragmentation, need for compaction off-line (downtime) or on-line Operating System Concepts – 9 th Edition 12. 14 Silberschatz, Galvin and Gagne © 2013

Contiguous Allocation n Mapping from logical to physical Q LA/512 R Block to be accessed = Q + starting address Displacement into block = R Operating System Concepts – 9 th Edition 12. 15 Silberschatz, Galvin and Gagne © 2013

Extent-Based Systems n Many newer file systems (i. e. , Veritas File System) use a modified contiguous allocation scheme n Extent-based file systems allocate disk blocks in extents n An extent is a contiguous block of disks l Extents are allocated for file allocation l A file consists of one or more extents Operating System Concepts – 9 th Edition 12. 16 Silberschatz, Galvin and Gagne © 2013

Allocation Methods - Linked n Linked allocation – each file a linked list of blocks l File ends at nil pointer l No external fragmentation l Each block contains pointer to next block l No compaction, external fragmentation l Free space management system called when new block needed l Improve efficiency by clustering blocks into groups but increases internal fragmentation l Reliability can be a problem l Locating a block can take many I/Os and disk seeks Operating System Concepts – 9 th Edition 12. 17 Silberschatz, Galvin and Gagne © 2013

Linked Allocation n Each file is a linked list of disk blocks: blocks may be scattered anywhere on the disk block = pointer n Mapping Q LA/512 R Block to be accessed is the Qth block in the linked chain of blocks representing the file. Displacement into block = R Operating System Concepts – 9 th Edition 12. 18 Silberschatz, Galvin and Gagne © 2013

Linked Allocation Operating System Concepts – 9 th Edition 12. 19 Silberschatz, Galvin and Gagne © 2013

Allocation Methods – Linked (Cont. ) n FAT (File Allocation Table) variation l Beginning of volume has table, indexed by block number l Much like a linked list, but faster on disk and cacheable l New block allocation simple Operating System Concepts – 9 th Edition 12. 20 Silberschatz, Galvin and Gagne © 2013

File-Allocation Table -1 Operating System Concepts – 9 th Edition 12. 21 Silberschatz, Galvin and Gagne © 2013

Allocation Methods - Indexed n Indexed allocation l Each file has its own index block(s) of pointers to its data blocks n Logical view Operating System Concepts – 9 th Edition 12. 22 Silberschatz, Galvin and Gagne © 2013

Example of Indexed Allocation Operating System Concepts – 9 th Edition 12. 23 Silberschatz, Galvin and Gagne © 2013

Indexed Allocation – Mapping (Cont. ) n How large the index block should be? How to allow for large files? l Linked scheme – we can link together several index blocks. For large files, the last address points to another index block rather than to a data block. l Multilevel index – a variant from the linked scheme. A first-level index block points to a set of a second-level index blocks which in turn point file blocks. It can continue to third or fourth index blocks as needed for file sizes. 4 Two-level index (e. g. , 4 K blocks could store 1, 024 four-byte pointers in outer index, each points to a block of pointers, each points to data blocks of 4 K size -> 1, 048, 567 data blocks and file size of up to 4 GB) Operating System Concepts – 9 th Edition 12. 24 Silberschatz, Galvin and Gagne © 2013

Indexed Allocation – Mapping (Cont. ) Operating System Concepts – 9 th Edition 12. 25 Silberschatz, Galvin and Gagne © 2013

Combined Scheme: UNIX UFS 4 K bytes per block, 32 -bit addresses More index blocks than can be addressed with 32 -bit file pointer Operating System Concepts – 9 th Edition 12. 26 Silberschatz, Galvin and Gagne © 2013

Performance n Best method depends on file access type l Contiguous great for sequential and random n Linked good for sequential, not random n Declare access type at creation -> select either contiguous or linked n Indexed more complex l Single block access could require 2 index block reads then data block read Operating System Concepts – 9 th Edition 12. 27 Silberschatz, Galvin and Gagne © 2013

Free-Space Management n File system maintains free-space list to track available blocks/clusters l (Using term “block” for simplicity) n Bit vector or bit map (n blocks) 0 1 2 n-1 … bit[i] = 1 block[i] free 0 block[i] occupied Block number calculation (number of bits per word) * (number of 0 -value words) + offset of first 1 bit CPUs have instructions to return offset within word of first “ 1” bit Operating System Concepts – 9 th Edition 12. 28 Silberschatz, Galvin and Gagne © 2013

Free-Space Management (Cont. ) n Bit map requires extra space l Example: block size = 4 KB = 212 bytes disk size = 240 bytes (1 terabyte) n = 240/212 = 228 bits (or 32 MB) if clusters of 4 blocks -> 8 MB of memory n Easy to get contiguous files Operating System Concepts – 9 th Edition 12. 29 Silberschatz, Galvin and Gagne © 2013

Linked Free Space List on Disk n Linked list (free list) l Cannot get contiguous space easily l No waste of space l No need to traverse the entire list (if # free blocks recorded) Operating System Concepts – 9 th Edition 12. 30 Silberschatz, Galvin and Gagne © 2013

Free-Space Management (Cont. ) n Grouping l Modify linked list to store address of next n-1 free blocks in first free block, plus a pointer to next block that contains free-blockpointers (like this one) n Counting l Because space is frequently contiguously used and freed, with contiguous-allocation, extents, or clustering 4 Keep address of first free block and count of following free blocks 4 Free space list then has entries containing addresses and counts Operating System Concepts – 9 th Edition 12. 31 Silberschatz, Galvin and Gagne © 2013

Performance n Performance l Keeping data and metadata close together l Buffer cache – separate section of main memory for frequently used blocks l Synchronous writes sometimes requested by apps or needed by OS 4 No buffering / caching – writes must hit disk before acknowledgement 4 Asynchronous l writes more common, buffer-able, faster Reads frequently slower than writes Operating System Concepts – 9 th Edition 12. 32 Silberschatz, Galvin and Gagne © 2013

Page Cache n A page caches pages rather than disk blocks using virtual memory techniques and addresses n Memory-mapped I/O uses a page cache n Routine I/O through the file system uses the buffer (disk) cache n This leads to the following figure Operating System Concepts – 9 th Edition 12. 33 Silberschatz, Galvin and Gagne © 2013

I/O Without a Unified Buffer Cache Operating System Concepts – 9 th Edition 12. 34 Silberschatz, Galvin and Gagne © 2013

Unified Buffer Cache n A unified buffer cache uses the same page cache to cache both memory-mapped pages and ordinary file system I/O to avoid double caching n But which caches get priority, and what replacement algorithms to use? n Free-behind (rather than LRU) and read-ahead – techniques to optimize sequential access Operating System Concepts – 9 th Edition 12. 35 Silberschatz, Galvin and Gagne © 2013

I/O Using a Unified Buffer Cache Operating System Concepts – 9 th Edition 12. 36 Silberschatz, Galvin and Gagne © 2013

Recovery n Consistency checking – compares data in directory structure with data blocks on disk, and tries to fix inconsistencies l Can be slow and sometimes fails n Use system programs to back up data from disk to another storage device (magnetic tape, other magnetic disk, optical) n Recover lost file or disk by restoring data from backup Operating System Concepts – 9 th Edition 12. 37 Silberschatz, Galvin and Gagne © 2013

Log Structured File Systems n Log structured (or journaling) file systems record each metadata update to the file system as a transaction n All transactions are written to a log l A transaction is considered committed once it is written to the log (sequentially) l Sometimes to a separate device or section of disk l However, the file system may not yet be updated n The transactions in the log are asynchronously written to the file system structures l When the file system structures are modified, the transaction is removed from the log n If the file system crashes, all remaining transactions in the log must still be performed n Faster recovery from crash, removes chance of inconsistency of metadata Operating System Concepts – 9 th Edition 12. 38 Silberschatz, Galvin and Gagne © 2013