Lecture 12 File Management File Management n provides

Lecture 12: File Management

File Management n provides the file abstraction for data storage n guarantees data validity (most of the time) n performance: throughput and response time n enforces protection Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 2 Silberschatz, Galvin and Gagne © 2005

File-System Interface n File Concept n Access Methods n Directory Structure n File-System Mounting n File Sharing n Protection Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 3 Silberschatz, Galvin and Gagne © 2005

File Concept n Contiguous logical address space n Persistent - stored on non-volatile memory (storage) n Identified by (external) name pathname in UNIX Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 4 Silberschatz, Galvin and Gagne © 2005

File Structure n None - sequence of words, bytes n Simple record structure Lines Fixed length Variable length n Complex Structures Formatted document Relocatable load file n Can simulate last two with first method by inserting appropriate control characters n Who decides: Operating system Program Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 5 Silberschatz, Galvin and Gagne © 2005

File Attributes n Name – only information kept in human-readable form n Identifier – unique tag (number) identifies file within file system n Type – needed for systems that support different types n Location – pointer to file location on device n Size – current file size n Protection – controls who can do reading, writing, executing n Time, date, and user identification – data for protection, security, and usage monitoring n Information about files are kept in the directory structure, which is maintained on the disk Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 6 Silberschatz, Galvin and Gagne © 2005

File Operations n Create n Write n Read n Reposition within file n Delete n Truncate n Open(Fi) –open read/write session for Fi File pointer: pointer to last read/write location, n Close (Fi) – close read/write session Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 7 Silberschatz, Galvin and Gagne © 2005

Access Methods Sequential Access read next write next reset no read after last write (rewrite) n Direct Access read n write n position to n read next write next rewrite n n = relative block number n Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 8 Silberschatz, Galvin and Gagne © 2005

Directories n Set of files n Used to organize files n Translates external file names to internal file names (file labels, i- nodes, file control blocks) n Treated as a file but with special operations Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 9 Silberschatz, Galvin and Gagne © 2005

Operations Performed on Directory n Search for a file n Create a file n Delete a file n List a directory n Rename a file n Traverse the file system Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 10 Silberschatz, Galvin and Gagne © 2005

Organize the Directory (Logically) to Obtain n Efficiency – locating a file quickly n Naming – convenient to users Two users can have same name for different files The same file can have several different names n Grouping – logical grouping of files by properties, (e. g. , all Java programs, all games, …) Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 11 Silberschatz, Galvin and Gagne © 2005

Single-Level Directory n A single directory for all users Naming problem Grouping problem Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 12 Silberschatz, Galvin and Gagne © 2005

Two-Level Directory n Separate directory for each user n Path name n Can have the same file name for different user n Efficient searching n No grouping capability Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 13 Silberschatz, Galvin and Gagne © 2005

Tree-Structured Directories Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 14 Silberschatz, Galvin and Gagne © 2005

Tree-Structured Directories (Cont) n Efficient searching n Grouping Capability n Current directory (working directory) cd /spell/mail/prog type list Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 15 Silberschatz, Galvin and Gagne © 2005

Tree-Structured Directories (Cont) n Absolute or relative path name n Creating a new file is done in current directory n Delete a file rm <file-name> n Creating a new subdirectory is done in current directory mkdir <dir-name> Example: if in current directory /mail mkdir count mail prog copy prt exp count Deleting “mail” deleting the entire subtree rooted by “mail” Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 16 Silberschatz, Galvin and Gagne © 2005

Acyclic-Graph Directories n Have shared subdirectories and files Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 17 Silberschatz, Galvin and Gagne © 2005

Acyclic-Graph Directories (Cont. ) n Two different names (aliasing) n If dict deletes list dangling pointer Solutions: Backpointers, so we can delete all pointers Variable size records a problem Backpointers using a daisy chain organization Entry-hold-count solution n New directory entry type Link – another name (pointer) to an existing file Resolve the link – follow pointer to locate the file Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 18 Silberschatz, Galvin and Gagne © 2005

File System Mounting n A file system must be mounted before it can be accessed n A file system is mounted at a mount point (a directory) Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 19 Silberschatz, Galvin and Gagne © 2005

(a) Existing. (b) Unmounted Partition Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 20 Silberschatz, Galvin and Gagne © 2005

Mount Point Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 21 Silberschatz, Galvin and Gagne © 2005

File Sharing n Sharing of files on multi-user systems is desirable n Sharing may be done through a protection scheme n On distributed systems, files may be shared across a network n Network File System (NFS) is a common distributed file-sharing method Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 22 Silberschatz, Galvin and Gagne © 2005

File Sharing – Multiple Users n User IDs identify users, allowing permissions and protections to be per-user n Group IDs allow users to be in groups, permitting group access rights Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 23 Silberschatz, Galvin and Gagne © 2005

File Sharing – Remote File Systems n Uses networking to allow file system access between systems n Manually via programs like FTP Automatically, seamlessly using distributed file systems Semi automatically via the world wide web Client-server model allows clients to mount remote file systems from servers Server can serve multiple clients Client and user-on-client identification is insecure or complicated NFS is standard UNIX client-server file sharing protocol CIFS is standard Windows protocol Standard operating system file calls are translated into remote calls Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 24 Silberschatz, Galvin and Gagne © 2005

File Sharing – Failure Modes n Remote file systems add new failure modes, due to network failure, server failure n Recovery from failure can involve state information about status of each remote request n Stateless protocols such as NFS include all information in each request, allowing easy recovery but less security Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 25 Silberschatz, Galvin and Gagne © 2005

File Sharing – Consistency Semantics n Consistency semantics specify how multiple users are to access a shared file simultaneously Similar to Ch 7 process synchronization algorithms 4 Tend to be less complex due to disk I/O and network latency (for remote file systems Andrew File System (AFS) implemented complex remote file sharing semantics Unix file system (UFS) implements: 4 Writes to an open file visible immediately to other users of the same open file 4 Sharing file pointer to allow multiple users to read and write concurrently AFS has session semantics 4 Writes only visible to sessions starting after the file is closed Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 26 Silberschatz, Galvin and Gagne © 2005

Protection n File owner/creator should be able to control: what can be done by whom n Types of access Read Write Execute Append Delete List Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 27 Silberschatz, Galvin and Gagne © 2005

Access Lists and Groups n Mode of access: read, write, execute n Three classes of users RWX a) owner access 7 111 RWX b) group access 6 110 RWX c) public access 1 001 n Ask manager to create a group (unique name), say G, and add some users to the group. n For a particular file (say game) or subdirectory, define an appropriate access. owner chmod group public 761 game Attach a group to a file chgrp Operating System Concepts – 7 th Edition, Jan 1, 2005 G game 10. 28 Silberschatz, Galvin and Gagne © 2005

Protection Mechanisms n n files are OS objects: unique names and a finite set of operations that processes can perform on them protection domain is a set of {object, rights}, where right is the permission to perform one of the operations n at every instant in time, each process runs in some protection domain n in Unix, a protection domain is {uid, gid} n n protection domain in Unix is switched when running a program with SETUID/SETGID set or when the process enters the kernel mode by issuing a system call how to store all the protection domains ? Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 29 Silberschatz, Galvin and Gagne © 2005

Protection Mechanisms (cont’d) n n Access Control List (ACL): associate with each object a list of all the protection domains that may access the object and how in Unix ACL is reduced to three protection domains: owner, group and others Capability List (C-list): associate with each process a list of objects that may be accessed along with the operations C-list implementation issues: where/how to store them (hardware, kernel, encrypted in user space) and how to revoke them Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 30 Silberschatz, Galvin and Gagne © 2005

A Sample UNIX Directory Listing Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 31 Silberschatz, Galvin and Gagne © 2005

Secondary Storage Management n Space must be allocated to files n Must keep track of the space available for allocation Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 32 Silberschatz, Galvin and Gagne © 2005

Preallocation n Need the maximum size for the file at the time of creation n Difficult to reliably estimate the maximum potential size of the file n Tend to overestimated file size to avoid running out of space Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 33 Silberschatz, Galvin and Gagne © 2005

Methods of File Allocation (1) n Contiguous allocation Single set of blocks is allocated to a file at the time of creation A single entry in the file allocation table 4 Starting block and length of the file n External fragmentation will occur Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 34 Silberschatz, Galvin and Gagne © 2005

Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 35 Silberschatz, Galvin and Gagne © 2005

Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 36 Silberschatz, Galvin and Gagne © 2005

Methods of File Allocation (2) n Chained allocation Allocation on basis of individual block Each block contains a pointer to the next block in the chain A single entry in the file allocation table 4 Starting block and length of file n No external fragmentation n Best for sequential files n No accommodation for the principle of locality Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 37 Silberschatz, Galvin and Gagne © 2005

Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 38 Silberschatz, Galvin and Gagne © 2005

Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 39 Silberschatz, Galvin and Gagne © 2005

Methods of File Allocation (3) n Indexed allocation File allocation table contains a separate one-level index for each file The index has one entry for each portion allocated to the file The file allocation table contains block number for the index Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 40 Silberschatz, Galvin and Gagne © 2005

Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 41 Silberschatz, Galvin and Gagne © 2005

Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 42 Silberschatz, Galvin and Gagne © 2005

File Allocation n contiguous: a contiguous set of blocks is allocated to a file at the time of file creation good for sequential files file size must be known at the time of file creation external fragmentation chained allocation: each block contains a pointer to the next one in the chain consolidation to improve locality indexed allocation: good both for sequential and direct access (UNIX) Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 43 Silberschatz, Galvin and Gagne © 2005

Free Space Management n bitmap: one bit for each block on the disk good to find a contiguous group of free blocks small enough to be kept in memory chained free portions: {pointer to the next one, length} index: treats free space as a file Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 44 Silberschatz, Galvin and Gagne © 2005

UNIX File System n Naming External/Internal names translation using directories n Lookup File blocks Disk blocks n Protection n Free Space Management Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 45 Silberschatz, Galvin and Gagne © 2005

File Naming n External names (used by the application) Pathname: /usr/users/file 1 n Internal names (used by the OS kernel) I-node: file number/index on disk File system on disk superblock 0 1 I-node area ( one I-node per file) Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 46 File-block area Silberschatz, Galvin and Gagne © 2005

Directories n Files that store translation tables (external names to internal names) usr usr users 23 users 41 file 1 87 Root directory (always I-node 2) /usr/users/file 1 corresponds to I-node 87 Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 47 Silberschatz, Galvin and Gagne © 2005

File Content Lookup n address table used to translate logical file blocks into disk blocks n address table stored in the I-node File with i-node 87 0 1 address table 2 File System disk Operating System Concepts – 7 th Edition, Jan 1, 2005 45 10. 48 65 45 65 85 85 Silberschatz, Galvin and Gagne © 2005

Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 49 Silberschatz, Galvin and Gagne © 2005

File Protection n ACL with three protection domains (file owner, file owner group, others) n Access rights: read/write/execute n Stored in the i-node Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 50 Silberschatz, Galvin and Gagne © 2005

Free Space Management n Free I-nodes Marked as free on disk An array of 50 free i-nodes stored in the superblock n Free file blocks Stored as a list of 50 - free block arrays First array stored in the superblock Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 51 Silberschatz, Galvin and Gagne © 2005

In-Kernel File System Data Structures Application OS Kernel fd=open(pathname, mode); for (. . ) read(fd, buf, size); close(fd); PCBs Per-process Open File Table /* fd = index in Per-Proc OFT */ I-node cache Per-OS Open File Table (offset in file, ptr to I-node) Buffer cache File system on disk 0 1 Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 52 Silberschatz, Galvin and Gagne © 2005

File System Metadata Consistency Problem n n file system uses the buffer cache for performance reasons two copies of a disk block (buffer cache, disk) -> consistency problem if the system crashes before all the modified blocks are written back to disk the problem is critical especially for the blocks that contain control information (meta-data): directory blocks, i-node, free-list Solution: write through meta-data blocks (expensive) or order of write-backs is important ordinary file data blocks written back periodically (sync) utility programs for checking block and directory consistency after crash Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 53 Silberschatz, Galvin and Gagne © 2005

File System Metadata Consistency Problem: Examples n Example 1: create a new file n Two updates: (1) allocate a free I-node; (2) create an entry in the directory (1) and (2) must be write-through (expensive) or (1) must be writtenback before (2) If (2) is written back first and a crash occurs before (1) is written back the directory structure is inconsistent and cannot be recovered Example 2: write a new block to a file Two updates: (1) allocate a free block; (2) update the address table of the I-node (1) and (2) must be write-through or (1) must be written-back before (2) If (2) is written back first and a crash occurs before (1) is written back the I-node structure is inconsistent and cannot be recovered Operating System Concepts – 7 th Edition, Jan 1, 2005 10. 54 Silberschatz, Galvin and Gagne © 2005