Chapter 11 File System Implementation Operating System Concepts

Chapter 11: File System Implementation Operating System Concepts– 8 th Edition Silberschatz, Galvin and Gagne © 2009

Chapter 11: File System Implementation n File-System Structure n File-System Implementation n Directory Implementation n Allocation Methods n Free-Space Management n Efficiency and Performance n Recovery Operating System Concepts – 8 th Edition 11. 2 Silberschatz, Galvin and Gagne © 2009

Objectives n To describe the details of implementing local file systems and directory structures n To describe the implementation of remote file systems n To discuss block allocation and free-block algorithms and trade-offs Operating System Concepts – 8 th Edition 11. 3 Silberschatz, Galvin and Gagne © 2009

FILE-SYSTEM STRUCTURE Operating System Concepts – 8 th Edition 11. 4 Silberschatz, Galvin and Gagne © 2009

File-System Structure n File structure l Logical storage unit l Collection of related information n File system resides on secondary storage (disks) l Provides user interface to storage, mapping logical to physical l Provides efficient and convenient access to disk by allowing data to be stored, located, and retrieved easily n Disk provides in-place rewrite and random access l I/O transfers performed in blocks of sectors (usually 512 bytes) n File control block (FCB) – storage structure consisting of information about a file n Device driver controls the physical device n File system organized into layers Operating System Concepts – 8 th Edition 11. 5 Silberschatz, Galvin and Gagne © 2009

I/O Control Layer n Device drivers manage I/O devices at the I/O control layer l Transfers information between main memory and the disk l Translator: given commands like “read drive 1, cylinder 72, track 2, sector 10, into memory location 1060” outputs low-level hardware specific commands to hardware controller Operating System Concepts – 8 th Edition 11. 6 Silberschatz, Galvin and Gagne © 2009

Basic File System Layer n Basic file system given command like “retrieve block 123” translates to device driver l Also manages memory buffers and caches (allocation, freeing, replacement) l Buffers hold data in transit l Caches hold frequently used data l Also known as physical I/O layer Operating System Concepts – 8 th Edition 11. 7 Silberschatz, Galvin and Gagne © 2009

File Organization Module Layer n File organization module understands files, logical address, and physical blocks l Translates logical block # to physical block # l Manages free space, disk allocation Operating System Concepts – 8 th Edition 11. 8 Silberschatz, Galvin and Gagne © 2009

Logical File System Layer n Logical file system manages metadata information l Translates file name into file number, file handle, location by maintaining file control blocks (inodes in Unix) l Directory management l Protection Operating System Concepts – 8 th Edition 11. 9 Silberschatz, Galvin and Gagne © 2009

File System Layers (Cont. ) n Layering useful for reducing complexity and redundancy, but adds overhead and can decrease performance n Logical layers can be implemented by any coding method according to OS designer n Many file systems, sometimes many within an operating system l Each with its own format (CD-ROM is ISO 9660; Unix has UFS, FFS; Windows has FAT, FAT 32, NTFS as well as floppy, CD, DVD Blu-ray, Linux has more than 40 types, with extended file system ext 2 and ext 3 leading; plus distributed file systems, etc. ) n New ones still arriving: ZFS, Google. FS, Oracle ASM, FUSE Operating System Concepts – 8 th Edition 11. 10 Silberschatz, Galvin and Gagne © 2009

FILE-SYSTEM IMPLEMENTATION Operating System Concepts – 8 th Edition 11. 11 Silberschatz, Galvin and Gagne © 2009

File-System Implementation n We have system calls at the API level, but how do we implement their functions? l On-disk structures 4 Boot control block 4 Volume control block 4 Directory 4 File l structure Control Block (FCB) In-memory structures 4 Mount table 4 Directory-structure 4 System-wide 4 Per-process cache open-file table 4 Buffers Operating System Concepts – 8 th Edition 11. 12 Silberschatz, Galvin and Gagne © 2009

On-Disk File System Structures n Boot control block contains info needed by the system to boot OS (if present) from that volume l Usually first block of volume n Volume control block contains volume details l Total # blocks, # free blocks, block size, free block pointers or array l UFS: superblock l NTFS: master file table n Directory structure organizes the files l UFS: Names and inode numbers l NTFS: part of master file table n Per-file File Control Block (FCB) contains many details about the file l UFS: Inode number, permissions, size, dates l NTFS: part of master file table Operating System Concepts – 8 th Edition 11. 13 Silberschatz, Galvin and Gagne © 2009

A Typical File Control Block Operating System Concepts – 8 th Edition 11. 14 Silberschatz, Galvin and Gagne © 2009

In-Memory File System Structures n Mount table contains information about each mounted volume n Directory-structure cache (recently accessed directories) n System-wide open-file table l FCB for each open file n Per-process open-file table l Pointers to system-wide table n Buffers l Hold data blocks from secondary storage Operating System Concepts – 8 th Edition 11. 15 Silberschatz, Galvin and Gagne © 2009

Creating A File n Application calls logical file system n Logical file system l Knows format of directory structure l Allocates FCB n System updates directory structure n open() returns a file handle for subsequent use Operating System Concepts – 8 th Edition 11. 16 Silberschatz, Galvin and Gagne © 2009

open() & read() (cache) Operating System Concepts – 8 th Edition 11. 17 Silberschatz, Galvin and Gagne © 2009

Partitions and Mounting n Partition can be a volume containing a file system (“cooked”) or raw (just a sequence of blocks with no file system) n Boot block can point to boot volume or boot loader set of blocks that contain enough code to know how to load the kernel from the file system l Or a boot management program for multi-OS booting n Root partition contains the OS, other partitions can hold other OSs, other file systems, or be raw l Mounted at boot time l Other partitions can mount automatically or manually n At mount time, file system consistency checked l Is all metadata correct? 4 If not, fix it, try again 4 If yes, add to mount table, allow access Operating System Concepts – 8 th Edition 11. 18 Silberschatz, Galvin and Gagne © 2009

Virtual File Systems n Virtual File Systems (VFS) on Unix provide an object-oriented way of implementing file systems l Alternative is to write directory and file routines for each file system n VFS allows the same system call interface (the API) to be used for different types of file systems l Separates file-system generic operations from implementation details l Implementation can be one of many file systems types, or network file system 4 Implements l vnodes which hold inodes or network file details Then dispatches operation to appropriate file system implementation routines n The API is to the VFS interface, rather than any specific type of file system Operating System Concepts – 8 th Edition 11. 19 Silberschatz, Galvin and Gagne © 2009

Schematic View of Virtual File System Operating System Concepts – 8 th Edition 11. 20 Silberschatz, Galvin and Gagne © 2009

Virtual File System Implementation n For example, Linux has four object types: l inode: represents a file l file: represents an open file l superblock: represents an entire file system l dentry: represents a directory entry n VFS defines set of operations on the objects that must be implemented l Every object has a pointer to a function table 4 Function table has addresses of routines to implement that function on that object Operating System Concepts – 8 th Edition 11. 21 Silberschatz, Galvin and Gagne © 2009

DIRECTORY IMPLEMENTATION Operating System Concepts – 8 th Edition 11. 22 Silberschatz, Galvin and Gagne © 2009

Directory Implementation 1. Linear list of file names with pointer to the data blocks l Simple to program l Time-consuming to execute 4 Linear search time 4 Could keep ordered alphabetically via linked list or use B+ tree 2. Hash Table – linear list with hash data structure l Decreases directory search time l Collisions – situations where two file names hash to the same location l Only good if entries are fixed size, or use chained-overflow method Operating System Concepts – 8 th Edition 11. 23 Silberschatz, Galvin and Gagne © 2009

ALLOCATION METHODS Operating System Concepts – 8 th Edition 11. 24 Silberschatz, Galvin and Gagne © 2009

Allocation Methods n An allocation method refers to how disk blocks are allocated for files l Contiguous allocation l Linked allocation l Indexed allocation n Performance considerations: l Sequential access l Direct access l Fragmentation l Other Operating System Concepts – 8 th Edition 11. 25 Silberschatz, Galvin and Gagne © 2009

Allocation Methods - Contiguous n Contiguous allocation – each file occupies a set of contiguous blocks Operating System Concepts – 8 th Edition 11. 26 l Pros? l Cons? Silberschatz, Galvin and Gagne © 2009

Allocation Methods - Contiguous n Contiguous allocation l Best performance in most cases l Simple – only starting location (block #) and length (number of blocks) are required l Problems include: 4 Finding 4 File space for file size at allocation time 4 External 4 Need fragmentation for compaction Operating System Concepts – 8 th Edition 11. 27 Silberschatz, Galvin and Gagne © 2009

Extent-Based Systems n Many newer file systems (i. e. , Veritas File System) use a modified contiguous allocation scheme n Extent-based file systems allocate disk blocks in extents n An extent is a contiguous block of disks l Extents are allocated for file allocation l A file consists of one or more extents Operating System Concepts – 8 th Edition 11. 28 Silberschatz, Galvin and Gagne © 2009

Allocation Methods - Linked n Linked allocation – each file is a linked list of blocks Operating System Concepts – 8 th Edition 11. 29 l Each block contains pointer to next block l File ends at null pointer Silberschatz, Galvin and Gagne © 2009

Allocation Methods - Linked n Linked allocation l No external fragmentation l Free space management system called when new block needed l Improve efficiency by clustering blocks into groups but increases internal fragmentation l Reliability can be a problem l Locating a block can take many I/Os and disk seeks n FAT (File Allocation Table) variation l Beginning of volume has table, indexed by block number l Much like a linked list, but faster on disk and cacheable l New block allocation simple Operating System Concepts – 8 th Edition 11. 30 Silberschatz, Galvin and Gagne © 2009

File-Allocation Table Operating System Concepts – 8 th Edition 11. 31 Silberschatz, Galvin and Gagne © 2009

Allocation Methods - Indexed n Indexed allocation l Operating System Concepts – 8 th Edition Each file has its own index block(s) of pointers to its data blocks 11. 32 Silberschatz, Galvin and Gagne © 2009

Allocation Methods - Indexed n Need index table n Random access n Dynamic access without external fragmentation, but have overhead of index block n What’s the maximum file size? l Block size / address size * block size l Linked scheme – Link blocks of index table (no limit on size) Operating System Concepts – 8 th Edition 11. 33 Silberschatz, Galvin and Gagne © 2009

Combined Scheme: UNIX UFS (4 K bytes per block, 32 -bit addresses) n How large of a file can be accessed with direct blocks? 12 * 4 KB = 48 KB n How large of a file can be accessed with single indirect? l (4 K/4) * 4 KB = 4 MB l (12) Operating System Concepts – 8 th Edition 11. 34 Silberschatz, Galvin and Gagne © 2009

Combined Scheme: UNIX UFS (4 K bytes per block, 32 -bit addresses) n How large of a file can be accessed with double indirect? (4 K/4)2 * 4 KB = 4 GB n How large of a file can be accessed with triple indirect? l (4 K/4)3 * 4 KB = 4 TB l (12) Operating System Concepts – 8 th Edition 11. 35 Silberschatz, Galvin and Gagne © 2009

Performance n Best method depends on file access type l Contiguous great for sequential and random n Linked is not good for random (but good for sequential) n Declare access type at creation -> select either contiguous or linked n Indexed more complex l Single block access could require 2 index block reads then data block read l Clustering can help improve throughput, reduce CPU overhead Operating System Concepts – 8 th Edition 11. 36 Silberschatz, Galvin and Gagne © 2009

Performance (Cont. ) n Adding instructions to the execution path to save one disk I/O is reasonable l Intel Core i 7 Extreme Edition 990 x (2011) at 3. 46 Ghz = 159, 000 MIPS 4 http: //en. wikipedia. org/wiki/Instructions_per_second l Typical disk drive at 250 I/Os per second 4 159, 000 l MIPS / 250 = 630 million instructions during one disk I/O Fast SSD drives provide 60, 000 I/Os per second 4 159, 000 MIPS / 60, 000 = 2. 65 millions instructions during one disk I/O Operating System Concepts – 8 th Edition 11. 37 Silberschatz, Galvin and Gagne © 2009

• Bit map • Linked list • Grouping • Counting • Space maps FREE-SPACE MANAGEMENT Operating System Concepts – 8 th Edition 11. 38 Silberschatz, Galvin and Gagne © 2009

Free-Space Management: Bit maps n File system maintains free-space list to track available blocks/clusters l (Using term “block” for simplicity) n Bit vector or bit map (n blocks) 0 1 2 n-1 ��� … bit[i] = 1 block[i] free 0 block[i] occupied CPUs have instructions to return offset within word of first “ 1” bit Operating System Concepts – 8 th Edition 11. 39 Silberschatz, Galvin and Gagne © 2009

Free-Space Management: Bit maps (Cont. ) n Bit map requires extra space l Example: block size = 4 KB = 212 bytes disk size = 240 bytes (1 terabyte) n = 240/212 = 228 bits (or 256 MB) if clusters of 4 blocks -> 64 MB of memory n Easy to get contiguous files Operating System Concepts – 8 th Edition 11. 40 Silberschatz, Galvin and Gagne © 2009

Linked Free Space List on Disk n Linked list (free list) l l l Each free block has the address of the next free block Location of the first entry kept in a special location Cannot get contiguous space easily No wasted space No need to traverse the entire list (if # free blocks recorded) FAT: Records free block info Operating System Concepts – 8 th Edition 11. 41 Silberschatz, Galvin and Gagne © 2009

Free-Space Management (Cont. ) n Grouping Linked list of blocks that are full of addresses of free blocks l Last entry is the address of the next “index” block l Less I/Os n Counting l Because space is frequently contiguously used and freed, with contiguous-allocation, extents, or clustering l Modify linked list l In addition to next address, store number of free blocks l Larger records, but you need fewer of them l Operating System Concepts – 8 th Edition 11. 42 Silberschatz, Galvin and Gagne © 2009

Free-Space Management (Cont. ) n Space Maps l Used in ZFS Consider meta-data I/O on very large file systems 4 Full data structures like bit maps couldn’t fit in memory 4 Thousands of I/Os to read info l Divides device space into metaslab units and manages metaslabs l 4 Given volume can contain hundreds of metaslabs l Each metaslab has associated space map 4 Uses counting algorithm l But records to log file rather than file system 4 Log of all block activity, in time order, in counting format l Metaslab activity -> load space map into memory in balanced-tree structure, indexed by offset 4 Replay log into that structure 4 Combine contiguous free blocks into single entry Silberschatz, Galvin and Gagne © 2009 11. 43 Operating System Concepts – 8 th Edition

EFFICIENCY AND PERFORMANCE Operating System Concepts – 8 th Edition 11. 44 Silberschatz, Galvin and Gagne © 2009

Efficiency n Efficiency dependent on: l Disk allocation and directory algorithms l Types of data kept in file’s directory entry l Pre-allocation or as-needed allocation of metadata structures 4 E. g. l UNIX inodes are preallocated Fixed-size or varying-size data structures Operating System Concepts – 8 th Edition 11. 45 Silberschatz, Galvin and Gagne © 2009

Performance n Performance l Keeping data and metadata close together l Buffer cache – separate section of main memory for frequently used blocks l Synchronous writes sometimes requested by apps or needed by OS 4 No buffering / caching – writes must hit disk before acknowledgement 4 Asynchronous writes more common, buffer-able, faster l Reads frequently slower than writes (because of asynchronous writes) l Free-behind and read-ahead – techniques to optimize sequential access Operating System Concepts – 8 th Edition 11. 46 Silberschatz, Galvin and Gagne © 2009

I/O Without a Unified Buffer Cache n A page caches pages rather than disk blocks using virtual memory techniques and addresses l Memory-mapped I/O uses a page cache n Routine I/O through the file system uses the buffer (disk) cache Operating System Concepts – 8 th Edition 11. 47 Silberschatz, Galvin and Gagne © 2009

I/O Using a Unified Buffer Cache n A unified buffer cache uses the same page cache to cache both memory-mapped pages and ordinary file system I/O l Avoids double caching l Some versions of UNIX and Linux n But which caches get priority, and what replacement algorithms to use? Operating System Concepts – 8 th Edition 11. 48 Silberschatz, Galvin and Gagne © 2009

RECOVERY Operating System Concepts – 8 th Edition 11. 49 Silberschatz, Galvin and Gagne © 2009

Recovery n Consistency checking – compares data in directory structure with data blocks on disk, and tries to fix inconsistencies l Can be slow and sometimes fails n Use system programs to back up data from disk to another storage device (magnetic tape, other magnetic disk, optical) n Recover lost file or disk by restoring data from backup n Backup example script: http: //www. cs. mtsu. edu/~hcarroll/3250/private/rsync. Backup. sh Operating System Concepts – 8 th Edition 11. 50 Silberschatz, Galvin and Gagne © 2009

Log Structured File Systems n Log structured (or journaling) file systems record each metadata update to the file system as a transaction n All transactions are written to a log l A transaction is considered committed once it is written to the log l Sometimes to a separate device or section of disk l However, the file system may not yet be updated n Log transactions are asynchronously written to the file system structures l When the file system structures are modified, the transaction is removed from the log n If the file system crashes, all remaining transactions in the log must still be performed n Faster recovery from crash, removes chance of inconsistency of metadata Operating System Concepts – 8 th Edition 11. 51 Silberschatz, Galvin and Gagne © 2009

Recap ① Give two reason why must the bit map for file allocation should be kept in secondary storage, rather than in main memory? ② Suppose that the pointer to the free-space list is lost. Can the system reconstruct the free-space list? Explain. ③ Consider a file system on a disk that has both logical and physical block sizes of 512 bytes. Assume that the information about each file is already in memory. How is the logical-to-physical address mapping accomplished for each of the three allocation strategies (contiguous, linked, and indexed)? (For the indexed allocation, assume that a file is always less than 512 blocks long. ) ④ Consider a file system that uses inodes to represent files. Disk blocks are 8 KB in size and a pointer to a disk block requires 4 bytes. This file system has 12 direct disk blocks, plus single, double, and triple indirect disk blocks. What is the maximum size of a file that can be stored in this file system? Operating System Concepts – 8 th Edition 11. 52 Silberschatz, Galvin and Gagne © 2009

Recap - Answers ① Give two reason why must the bit map for file allocation should be kept in secondary storage, rather than in main memory? l So that the free-space list would not be lost after a system crash l It may be too large to fit in main memory. ② Suppose that the pointer to the free-space list is lost. Can the system reconstruct the free-space list? Explain. l Must do “garbage collection”: search entire directory structure to determine which pages are already allocated to jobs. Those remaining unallocated pages could be relinked as the free-space list. Operating System Concepts – 8 th Edition 11. 53 Silberschatz, Galvin and Gagne © 2009

Recap - Answers ③ …logical and physical block sizes of 512 bytes … info … in memory. How is the logical-to-physical address mapping accomplished …? l Let Z be the starting file address (block number). l Contiguous 4 Divide logical address by 512; X = quotient Y = remainder 4 Physical l Linked 4 Divide 4 Phy. l block number: X + Z; Displacement into block: Y logical address by 508; X = quotient Y = remainder blk: Follow X links; Displacement: Y + 1 Indexed: 4 Divide 4 Phy. logical address by 512; X = quotient Y = remainder blk: Read index block, Xth location; Displacement: Y Operating System Concepts – 8 th Edition 11. 54 Silberschatz, Galvin and Gagne © 2009

Recap - Answers ④ Consider a file system that uses inodes to represent files. Disk blocks are 8 KB in size and a pointer to a disk block requires 4 bytes. This file system has 12 direct disk blocks, plus single, double, and triple indirect disk blocks. What is the maximum size of a file that can be stored in this file system? l Use all four address schemes: l Blocks addressable from 1 page: 8 KB/4 B = 211 l Direct: (12 * 8 KB) = 96 KB l Single: (211 * 8 KB) = 16 MB l Double: ((211)2 * 8 KB) = 32 GB l Triple: ((211)3 * 8 KB) = 64 TB l Max file size = sum Operating System Concepts – 8 th Edition 11. 55 Silberschatz, Galvin and Gagne © 2009

End of Chapter 11 Operating System Concepts– 8 th Edition Silberschatz, Galvin and Gagne © 2009