Chapter 11 File System Implementation Chapter 11 File
- Slides: 75
Chapter 11: File System Implementation
Chapter 11: File System Implementation n File-System Structure n File-System Implementation n Directory Implementation n Allocation Methods n Free-Space Management n Efficiency and Performance n Recovery n Log-Structured File Systems n NFS n Example: WAFL File System Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 2 Silberschatz, Galvin and Gagne © 2005
Objectives n To describe the details of implementing local file systems and directory structures n To describe the implementation of remote file systems n To discuss block allocation and free-block algorithms and trade-offs Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 3 Silberschatz, Galvin and Gagne © 2005
File-System Structure n File structure l Logical storage unit l Collection of related information n File system resides on secondary storage (disks) n File system organized into layers n File control block – storage structure consisting of information about a file n Issues in file system implementation : l Disk space allocation l Free space recovery l Tracking location of data l Performance Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 4 Silberschatz, Galvin and Gagne © 2005
Layered File System Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 5 Silberschatz, Galvin and Gagne © 2005
File system Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 6 Silberschatz, Galvin and Gagne © 2005
Devices n Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 7 Silberschatz, Galvin and Gagne © 2005
I/O Control n It consists of device drivers and interrupt handlers to transfer information between the main memory and the disk system. n Each controller card ( device ) on a system has a different set of addresses ( registers, a. k. a. ports ) that it listens to, and a unique set of command codes and results codes that it understands Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 8 Silberschatz, Galvin and Gagne © 2005
Basic File System n It issues generic commands to the appropriate device drivers to read and write physical blocks on the disk, without any consideration for what is in each block l E. g. get block 123 n Each physical block is identified by its numeric disk address( drive 1, cylinder 3, track 2, sector 10) Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 9 Silberschatz, Galvin and Gagne © 2005
File organization module n It knows about files and their logical blocks as well as physical blocks. n It can translate logical block addresses to physical block addresses for the basic file system. n the file organization module also maintains the list of free blocks, and allocates free blocks to files as needed. Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 10 10 Silberschatz, Galvin and Gagne © 2005
Logical File System n It manages metadata information (FCB) and the mapping from filename to FCB. Metadata: It includes all of the file system structure except the actual data. n It manages the directory structure to provide the file organization information with the information. Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 11 11 Silberschatz, Galvin and Gagne © 2005
A Typical File Control Block Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 12 Silberschatz, Galvin and Gagne © 2005
File-System Implementation n On-disk l Boot block (per partition) l Superblock (per partition) l Directory structure (per partition ) l File control block / Inode (per file) n In-memory l Mount table l Directory structure cache l The open-file table (system-wide and per process) l Buffers of the file-system blocks Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 13 Silberschatz, Galvin and Gagne © 2005
File System On Disk n MBR : Master boot record l Is the sector 0 in disk which is used for booting the computer. n Partition table : l has Operating System Concepts – 7 th information about the active partitions. Edition, Jan 1, 2005 11. 14 14 14 Silberschatz, Galvin and Gagne © 2005
n Boot block: l It contains information needed by the system to boot an operating system. n Super block contains important information about file system. l the total size of the partition, the block size, the inode number of the root directory, free block management, magic number, etc Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 15 Silberschatz, Galvin and Gagne © 2005
data structures stored in memory n An in-memory mount table. n An in-memory directory cache of recently accessed directory information. n A system-wide open file table, l File location on the disk l File size � l File open count (the number of processes using this file) n A per-process open file table, l Position pointer l Access rights l l Accounting Pointer to the system-wide open-file table entry Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 16 Silberschatz, Galvin and Gagne © 2005
Structures Figure 12 -3(a) refers to opening a file. Figure 12 -3(b) 11. 17 refers to reading a file. Silberschatz, Galvin and Gagne © 2005 Operating System Concepts – 7 th Edition, Jan 1, 2005
In-Memory File System : Open File n The open() passes a file name to the logical file system. n The open() first searches the system-wide open-file: if the file is already in use by another process. l If yes: 4 per-process table is adjusted to point to the existing entry in the system-wide table l If no: 4 the directory structure is searched for the given file name. 4 Once the file is found, the FCB is copied into a system-wide open-file table in memory. Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 18 Silberschatz, Galvin and Gagne © 2005
n An entry is added to the per-process open file table referencing the system-wide table, n A Counter in the System-Wide Table is incremented n An index into the per-process table is returned by the open( ) system call. n UNIX refers to this index as a file descriptor, and Windows refers to it as a file handle. Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 19 Silberschatz, Galvin and Gagne © 2005
In-Memory File System : Read File n The open() returns a pointer to the appropriate entry in the per process file-system table. n All file operations are then performed via this pointer Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 20 Silberschatz, Galvin and Gagne © 2005
In-Memory File System : Close File n The per-process table entry is removed. n The system-wide entry’s open count is decremented. n When all users that have opened the file close it, any updated metadata is copied back to the disk-based directory structure n the system-wide open-file table entry is removed. Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 21 Silberschatz, Galvin and Gagne © 2005
Virtual File Systems n Virtual File Systems (VFS) provide an object-oriented way of implementing file systems. n VFS allows the same system call interface (the API) to be used for different types of file systems. n The API is to the VFS interface, rather than any specific type of file system. Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 22 Silberschatz, Galvin and Gagne © 2005
Schematic View of Virtual File System Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 23 Silberschatz, Galvin and Gagne © 2005
Allocation Methods n An allocation method refers to how disk blocks are allocated for files: n Allocation Method : l Contiguous l Linked allocation l Indexed allocation Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 24 Silberschatz, Galvin and Gagne © 2005
Contiguous Allocation n Each file occupies a set of contiguous blocks on the disk n Simple – only starting location (block #) and length (number of blocks) are required n Random access n Wasteful of space (dynamic storage-allocation problem) n Files cannot grow Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 25 Silberschatz, Galvin and Gagne © 2005
Contiguous Allocation of Disk Space Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 26 Silberschatz, Galvin and Gagne © 2005
Extent-Based Systems n Many newer file systems (I. e. Veritas File System) use a modified contiguous allocation scheme n Extent-based file systems allocate disk blocks in extents n An extent is a contiguous block of disks l Extents are allocated for file allocation l A file consists of one or more extents. Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 27 Silberschatz, Galvin and Gagne © 2005
Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 28 Silberschatz, Galvin and Gagne © 2005
Linked Allocation n Each file is a linked list of disk blocks: blocks may be scattered anywhere on the disk. n Simple – need only starting address n Free-space management system – no waste of space n No random access n Ex : File-allocation table (FAT) – disk-space allocation used by MS- DOS and OS/2. block Operating System Concepts – 7 th Edition, Jan 1, 2005 = pointer 11. 29 Silberschatz, Galvin and Gagne © 2005
Linked Allocation Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 30 Silberschatz, Galvin and Gagne © 2005
File-Allocation Table Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 31 Silberschatz, Galvin and Gagne © 2005
Indexed Allocation n Brings all pointers together into the index block. n Logical view. index table Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 32 Silberschatz, Galvin and Gagne © 2005
Example of Indexed Allocation Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 33 Silberschatz, Galvin and Gagne © 2005
Indexed Allocation (Cont. ) n Need index table n Random access n Dynamic access without external fragmentation, but have overhead of index block. Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 34 Silberschatz, Galvin and Gagne © 2005
Indexed Allocation – Mapping (Cont. ) outer-index table Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 35 file Silberschatz, Galvin and Gagne © 2005
Combined Scheme: UNIX (4 K bytes per block) Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 36 Silberschatz, Galvin and Gagne © 2005
Free-Space Management n Since disk space is limited, we need to reuse the space from deleted files for new files, if possible. n To keep track of free disk space, the system maintains a free-space list. The free-space list records all free disk blocks. n To create a file, we search the free-space list for the required amount of space and allocate that space to the new file. n When a file is deleted, its disk space is added to the free-space list. Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 37 Silberschatz, Galvin and Gagne © 2005
n There are 4 methods for file management l Bit Vector l Linked List l Grouping l Counting Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 38 Silberschatz, Galvin and Gagne © 2005
Bit Vector n Bit vector (n blocks) 0 1 2 n-1 … bit[i] = 0 block[i] free 1 block[i] occupied Misal : We have blocks from 0 to 27 Free block : 2, 3, 4, 5, 8, 9, 10, 11, 12, 13, 17, 18, 25, 26, 27 Maka : 1100001100 …. Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 39 Silberschatz, Galvin and Gagne © 2005
n Bit vector requires extra space l Example: block size = 212 bytes disk size = 230 bytes (1 gigabyte) n = 230/212 = 218 bits (or 32 K bytes) n Need to protect: l Bit vector 4 Must be kept on disk 4 Copy in memory and disk may differ 4 Cannot allow for block[i] to have a situation where bit[i] = 1 in memory and bit[i] = 0 on disk Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 40 Silberschatz, Galvin and Gagne © 2005
Free-Space Management (Cont. ) n Solution: Set bit[i] = 1 in disk l Allocate block[i] l Set bit[i] = 1 in memory l n For efficiency, the bit vector is kept in memory and disk n If the disk is small, no problem; but when the disk size is big, it is impossible to keep in memory Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 41 Silberschatz, Galvin and Gagne © 2005
Linked List n Linked list (free list) l Cannot get contiguous space easily l No waste of space Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 42 Silberschatz, Galvin and Gagne © 2005
Grouping n Another approach is to store the addresses of n free blocks in the first free block. The last block contains the addresses of other n free blocks and so on. n If we have free block : 2, 3, 4, 5, 8, 9, 10, 11, 12, 13, 17, 18, 25, 26, 27 n Then we have : Group with N=3 l Block 2 : 3, 4, 5, l Block 5 : 8, 9, 10 l Block 10 : 11, 12 , 13 Operating System Concepts – 7 th Edition, Jan 1, 2005 Block 13 : 17, 18, 25 11. 43 43 Silberschatz, Galvin and Gagne © 2005
Counting n In this technique, instead of keeping a list of addresses of n free blocks, it is more convenient to keep the address of the first free blocks and the number n of free contiguous blocks that follow the first block. n If we have free block : 2, 3, 4, 5, 8, 9, 10, 11, 12, 13, 17, 18, 25, 26, 27 n Counting : l because free block is 2, 3, 4, 5 then 2, 4 l because free block is 8, 9, 10, 11, 12, 13 then 8, 6 l because free block is 25, 26, 27 then 25, 3 Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 44 44 Silberschatz, Galvin and Gagne © 2005
Free space Management Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 45 Silberschatz, Galvin and Gagne © 2005
Directory Implementation n Linear list of file names with pointer to the data blocks. l simple to program l time-consuming to execute n Hash Table – linear list with hash data structure. l decreases directory search time l collisions – situations where two file names hash to the same location l fixed size Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 46 46 Silberschatz, Galvin and Gagne © 2005
Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 47 Silberschatz, Galvin and Gagne © 2005
Collision n Hash collision resolved by separate chaining Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 48 Silberschatz, Galvin and Gagne © 2005
Efficiency and Performance n Efficiency dependent on: l disk allocation and directory algorithms l types of data kept in file’s directory entry n Performance l disk cache – separate section of main memory for frequently used blocks l free-behind and read-ahead – techniques to optimize sequential access l improve PC performance by dedicating section of memory as virtual disk, or RAM disk Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 49 Silberschatz, Galvin and Gagne © 2005
Page Cache n A page caches pages rather than disk blocks using virtual memory techniques n Memory-mapped I/O uses a page cache n Routine I/O through the file system uses the buffer (disk) cache n This leads to the following figure Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 50 Silberschatz, Galvin and Gagne © 2005
I/O Without a Unified Buffer Cache Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 51 Silberschatz, Galvin and Gagne © 2005
Unified Buffer Cache n A unified buffer cache uses the same page cache to cache both memory-mapped pages and ordinary file system I/O Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 52 Silberschatz, Galvin and Gagne © 2005
I/O Using a Unified Buffer Cache Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 53 Silberschatz, Galvin and Gagne © 2005
Recovery n Consistency checking – compares data in directory structure with data blocks on disk, and tries to fix inconsistencies n Use system programs to back up data from disk to another storage device (floppy disk, magnetic tape, other magnetic disk, optical) n Recover lost file or disk by restoring data from backup Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 54 Silberschatz, Galvin and Gagne © 2005
Log Structured File Systems n Log structured (or journaling) file systems record each update to the file system as a transaction n All transactions are written to a log l A transaction is considered committed once it is written to the log l However, the file system may not yet be updated n The transactions in the log are asynchronously written to the file system l When the file system is modified, the transaction is removed from the log n If the file system crashes, all remaining transactions in the log must still be performed Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 55 Silberschatz, Galvin and Gagne © 2005
The Sun Network File System (NFS) n An implementation and a specification of a software system for accessing remote files across LANs (or WANs) n The implementation is part of the Solaris and Sun. OS operating systems running on Sun workstations using an unreliable datagram protocol (UDP/IP protocol and Ethernet Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 56 Silberschatz, Galvin and Gagne © 2005
NFS (Cont. ) n Interconnected workstations viewed as a set of independent machines with independent file systems, which allows sharing among these file systems in a transparent manner l A remote directory is mounted over a local file system directory 4 The mounted directory looks like an integral subtree of the local file system, replacing the subtree descending from the local directory l Specification of the remote directory for the mount operation is nontransparent; the host name of the remote directory has to be provided 4 Files in the remote directory can then be accessed in a transparent manner l Subject to access-rights accreditation, potentially any file system (or directory within a file system), can be mounted remotely on top of any local directory Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 57 Silberschatz, Galvin and Gagne © 2005
NFS (Cont. ) n NFS is designed to operate in a heterogeneous environment of different machines, operating systems, and network architectures; the NFS specifications independent of these media n This independence is achieved through the use of RPC primitives built on top of an External Data Representation (XDR) protocol used between two implementation-independent interfaces n The NFS specification distinguishes between the services provided by a mount mechanism and the actual remote-file-access services Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 58 Silberschatz, Galvin and Gagne © 2005
Three Independent File Systems Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 59 Silberschatz, Galvin and Gagne © 2005
Mounting in NFS Mounts Operating System Concepts – 7 th Edition, Jan 1, 2005 Cascading mounts 11. 60 Silberschatz, Galvin and Gagne © 2005
NFS Mount Protocol n Establishes initial logical connection between server and client n Mount operation includes name of remote directory to be mounted and name of server machine storing it l Mount request is mapped to corresponding RPC and forwarded to mount server running on server machine l Export list – specifies local file systems that server exports for mounting, along with names of machines that are permitted to mount them n Following a mount request that conforms to its export list, the server returns a file handle—a key for further accesses n File handle – a file-system identifier, and an inode number to identify the mounted directory within the exported file system n The mount operation changes only the user’s view and does not affect the server side Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 61 Silberschatz, Galvin and Gagne © 2005
NFS Protocol n Provides a set of remote procedure calls for remote file operations. The procedures support the following operations: l searching for a file within a directory l reading a set of directory entries l manipulating links and directories l accessing file attributes l reading and writing files n NFS servers are stateless; each request has to provide a full set of arguments (NFS V 4 is just coming available – very different, stateful) n Modified data must be committed to the server’s disk before results are returned to the client (lose advantages of caching) n The NFS protocol does not provide concurrency-control mechanisms Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 62 Silberschatz, Galvin and Gagne © 2005
Three Major Layers of NFS Architecture n UNIX file-system interface (based on the open, read, write, and close calls, and file descriptors) n Virtual File System (VFS) layer – distinguishes local files from remote ones, and local files are further distinguished according to their file-system types l The VFS activates file-system-specific operations to handle local requests according to their file-system types l Calls the NFS protocol procedures for remote requests n NFS service layer – bottom layer of the architecture l Implements the NFS protocol Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 63 Silberschatz, Galvin and Gagne © 2005
Schematic View of NFS Architecture Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 64 Silberschatz, Galvin and Gagne © 2005
NFS Path-Name Translation n Performed by breaking the path into component names and performing a separate NFS lookup call for every pair of component name and directory vnode n To make lookup faster, a directory name lookup cache on the client’s side holds the vnodes for remote directory names Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 65 Silberschatz, Galvin and Gagne © 2005
NFS Remote Operations n Nearly one-to-one correspondence between regular UNIX system calls and the NFS protocol RPCs (except opening and closing files) n NFS adheres to the remote-service paradigm, but employs buffering and caching techniques for the sake of performance n File-blocks cache – when a file is opened, the kernel checks with the remote server whether to fetch or revalidate the cached attributes l Cached file blocks are used only if the corresponding cached attributes are up to date n File-attribute cache – the attribute cache is updated whenever new attributes arrive from the server n Clients do not free delayed-write blocks until the server confirms that the data have been written to disk Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 66 Silberschatz, Galvin and Gagne © 2005
Example: WAFL File System n Used on Network Appliance “Filers” – distributed file system appliances n “Write-anywhere file layout” n Serves up NFS, CIFS, http, ftp n Random I/O optimized, write optimized l NVRAM for write caching n Similar to Berkeley Fast File System, with extensive modifications Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 67 Silberschatz, Galvin and Gagne © 2005
The WAFL File Layout Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 68 Silberschatz, Galvin and Gagne © 2005
Snapshots in WAFL Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 69 Silberschatz, Galvin and Gagne © 2005
11. 02 Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 70 Silberschatz, Galvin and Gagne © 2005
End of Chapter 11
n http: //greenteapress. com/thinkos/html/thinkos 005. html n http: //www. cs. uic. edu/~jbell/Course. Notes/Operating. Systems/11_Fil e. System. Implementation. html Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 72 Silberschatz, Galvin and Gagne © 2005
Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 73 Silberschatz, Galvin and Gagne © 2005
Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 74 Silberschatz, Galvin and Gagne © 2005
Operating System Concepts – 7 th Edition, Jan 1, 2005 11. 75 Silberschatz, Galvin and Gagne © 2005
- Distributed file system implementation
- File-file yang dibuat oleh user pada jenis file di linux
- Remote file access in distributed file system
- In a file-oriented information system, a transaction file
- File system modules in distributed system
- Flat file service
- File system in operating system
- File system in operating system
- File system in operating system
- System implementation and maintenance
- Implementation of process in operating system
- System design implementation and operation
- A transaction is any business event that generates
- Transaction processing system architecture
- System construction
- The most creative and challenging phase of sdlc is
- Cse 132
- Rpc implementation in distributed system
- It interaction model
- Database systems: design, implementation, and management
- Oceannodes
- Job order contracting (joc) system implementation
- Database systems design implementation and management
- Difference between logical file and physical file
- Fungsi dari create file pada operasi-operasi file (cont.)
- An html file is a text file containing small markup tags.
- Chapter 6 implementation and evaluation
- Open closed isolated system
- Digestive respiratory and circulatory system
- Zfs distributed file system
- Namei algorithm in unix
- Coda file system
- Akses ke file system dilakukan melalui
- Os file management
- Sun nfs
- Frangipani: a scalable distributed file system
- How does a coda work
- Arsitektur pengelolaan file
- Irt 5433
- Union file system
- What is hierarchical file system
- Hadoop distributed file system
- File system reliability
- File based system adalah
- Fast file system
- Union file system
- E office file management system
- Replicated file system
- Coda file system architecture
- Distributed file system design
- What is coda file system
- File system maintenance
- Fat file system
- File based system
- Ceph: a scalable, high-performance distributed file system
- Ceph: a scalable, high-performance distributed file system
- Ceph scalability
- Hrfs file system
- Unix file layout
- Stateless vs stateful server
- Steganographic file system
- Shadow master in gfs
- Google file system architecture
- Signature file structure in information retrieval system
- Wafl file system
- Pengertian file system
- File-system interface
- Cluster based distributed file system
- Distributed file system
- Coda file system
- Evolution of file system data processing
- Distributed file system in cloud computing
- Ialloc algorithm in unix
- Lustre architecture
- Gfs google file system
- Android manifest persistent