Storage Jeff Chase Duke University Storage The Big

Storage Jeff Chase Duke University

Storage: The Big Issues 1. Disks are rotational media with mechanical arms. • High access cost caching and prefetching • Cost depends on previous access careful block placement and scheduling. 2. Stored data is hard state. • Stored data persists after a restart. • Data corruption and poor allocations also persist. • Allocate for longevity, and write carefully. 3. Disks fail. • Plan for failure redundancy and replication. 1. RAID: integrate redundancy with striping across multiple disks for higher throughput.

Rotational Media Track Sector Arm Cylinder Head Platter Access time = seek time + rotational delay + transfer time seek time = 5 -15 milliseconds to move the disk arm and settle on a cylinder rotational delay = 8 milliseconds for full rotation at 7200 RPM: average delay = 4 ms transfer time = 1 millisecond for an 8 KB block at 8 MB/s Bandwidth utilization is less than 50% for any noncontiguous access at a block grain.

Disks and Drivers • Disk hardware and driver software provide foundational support for block devices. • OS views the block devices as a collection of volumes. – A logical volume may be a partition of a single disk or a concatenation of multiple physical disks (e. g. , RAID). • volume == LUN • Each volume is an array of fixed-size sectors. – Name sector/block by (volume. ID, sector ID). – Read/write operations DMA data to/from physical memory. • Device interrupts OS on I/O completion. – ISR wakes process, updates internal records, etc.

RAID • Raid levels 3 through 5 • Backup and parity drives are shaded

Filesystems • Files – Sequentially numbered bytes or logical blocks. – Metadata stored in on-disk data object • e. g, Unix “inode” • Directories – A special kind of file with a set of name mappings. • E. g. , name to inode – Pointer to parent in rooted hierarchy: . . , / • System calls – Unix: open, close, read, write, stat, seek, sync, link, unlink, symlink, chdir, chroot, mount, chmod, chown.

A Typical Unix File Tree Each volume is a set of directories and files; a host’s file tree is the set of directories and files visible to processes on a given host. / File trees are built by grafting volumes from different volumes or from network servers. In Unix, the graft operation is the privileged mount system call, and each volume is a filesystem. bin ls mount point etc tmp sh project packages (volume root) mount (covered. Dir, volume) covered. Dir: directory pathname tex volume: device specifier or network volume root contents become visible at pathname covered. Dir emacs usr vmunix users

Abstractions User view Addressbook, record for Duke CPS Application addrfile ->fid, byte range* fid bytes block# File System device, block # Disk Subsystem surface, cylinder, sector

Directories wind: 18 0 directory inode snow: 62 0 rain: 32 hail: 48 sector 32 Entries or slots are found by a linear scan.

A Filesystem On Disk sector 0 sector 1 allocation bitmap file wind: 18 0 directory file 111000101101 10111101 snow: 62 0 once upo n a time /n in a l 10011010 001100010101 00101110 00011001 0100 rain: 32 hail: 48 and far away , lived th This is just an example (Nachos)

UNIX File System Calls Open files are named to by an integer file descriptor. char buf[BUFSIZE]; int fd; if ((fd = open(“. . /zot”, O_TRUNC | O_RDWR) == -1) { perror(“open failed”); exit(1); } while(read(0, buf, BUFSIZE)) { if (write(fd, buf, BUFSIZE) != BUFSIZE) { perror(“write failed”); exit(1); } } Standard descriptors (0, 1, 2) for input, output, error messages (stdin, stdout, stderr). Pathnames may be relative to process current directory. Process passes status back to parent on exit, to report success/failure. Process does not specify current file offset: the system remembers it.

File Sharing Between Parent/Child (UNIX) main(int argc, char *argv[]) { char c; int fdrd, fdwt; if ((fdrd = open(argv[1], O_RDONLY)) == -1) exit(1); if ((fdwt = creat([argv[2], 0666)) == -1) exit(1); fork(); for (; ; ) { } } if (read(fdrd, &c, 1) != 1) exit(0); write(fdwt, &c, 1); [Bach]

Operations on Directories (UNIX) • • • Link - make entry pointing to file Unlink - remove entry pointing to file Rename Mkdir - create a directory Rmdir - remove a directory

Access Control for Files • Access control lists - detailed list attached to file of users allowed (denied) access, including kind of access allowed/denied. • UNIX RWX - owner, group, everyone

File Systems: The Big Issues • • • Buffering disk data for access from the processor. – Block I/O (DMA) needs aligned physical buffers. – Block update is a read-modify-write. Creating/representing/destroying independent files. Allocating disk blocks and scheduling disk operations to deliver the best performance for the I/O stream. – What are the patterns in the request stream? Multiple levels of name translation. – Pathname inode, logical physical block Reliability and the handling of updates.

Representing a File On Disk file attributes: may include owner, access control list, time of create/modify/access, etc. once upo n a time /nin a l logical block 0 and far away , /nlived t logical block 1 block map physical block pointers in the block map are sector IDs or physical block numbers inode he wise and sage wizard. logical block 2

Meta-Data • File size • File type • Protection - access control information • History: creation time, last modification, last access. • Location of file which device • Location of individual blocks of the file on disk. • Owner of file • Group(s) of users associated with file

Representing Large Files inode Classical Unix Each file system block is a clump of sectors (4 KB, 8 KB, 16 KB). direct block map indirect block Inode == 128 bytes, packed into blocks. Each inode has 68 bytes of attributes and 15 block map entries. double indirect block Suppose block size = 8 KB 12 direct block map entries in the inode can map 96 KB of data. One indirect block (referenced by the inode) can map 16 MB of data. One double indirect block pointer in inode maps 2 K indirect blocks. maximum file size is 96 KB + 16 MB + (2 K*16 MB) +. . .

Unix index blocks • • Intuition – Many files are small • Length = 0, length = 1, length < 80, . . . – Some files are huge (3 gigabytes) “Clever heuristic” in Unix FFS inode – 12 (direct) block pointers: 12 * 8 KB = 96 KB • Availability is “free” - you need inode to open() file anyway – 3 indirect block pointers • single, double, triple

Unix index blocks

Unix index blocks

Unix index blocks

Unix index blocks

Unix index blocks

Links usr ln -s /usr/Marty/bar Lynn creat foo unlink foo Marty creat bar ln /usr/Lynn/foo bar unlink bar

Unix File Naming (Hard Links) directory A A Unix file may have multiple names. Each directory entry naming the file is called a hard link. Each inode contains a reference count showing how many hard links name it. directory B 0 wind: 18 rain: 32 0 hail: 48 sleet: 48 inode link count = 2 inode 48 link system call link (existing name, new name) create a new name for an existing file increment inode link count unlink system call (“remove”) unlink(name) destroy directory entry decrement inode link count if count == 0 and file is not in active use free blocks (recursively) and on-disk inode Illustrates: garbage collection by reference counting.

Unix Symbolic (Soft) Links A soft link is a file containing a pathname of some other file. directory A directory B 0 rain: 32 wind: 18 0 hail: 48 sleet: 67 symlink system call symlink (existing name, new name) allocate a new file (inode) with type symlink initialize file contents with existing name create directory entry for new file with new name inode link count = 1 . . /A/hail/0 inode 48 inode 67 The target of the link may be removed at any time, leaving a dangling reference. How should the kernel handle recursive soft links?

Filesystems • Each file volume (filesystem) has a type, determined by its disk layout or the network protocol used to access it. – ufs (ffs), lfs, nfs, rfs, cdfs, etc. – Filesystems are administered independently. • Modern systems also include “logical” pseudo-filesystems in the naming tree, accessible through the file syscalls. – procfs: the /proc filesystem allows access to process internals. – mfs: the memory file system is a memory-based scratch store. • Processes access filesystems through common syscalls

VFS: the Filesystem Switch • Sun Microsystems introduced the virtual file system interface in 1985 to accommodate diverse filesystem types cleanly. – VFS allows diverse specific file systems to coexist in a file tree, isolating all FS-dependencies in pluggable filesystem modules. user space syscall layer (file, uio, etc. ) network protocol stack (TCP/IP) Virtual File System (VFS) NFS FFS LFS *FS etc. device drivers

Vnodes • In the VFS framework, every file or directory in active use is represented by a vnode object in kernel memory. free vnodes syscall layer Each vnode has a standard file attributes struct. Generic vnode points at filesystem-specific struct (e. g. , inode, rnode), seen only by the filesystem. Each specific file system maintains a cache of its resident vnodes. NFS UFS Vnode operations are macros that vector to filesystem-specific procedures.

V/Inode Cache HASH(fsid, fileid) VFS free list head Active vnodes are reference- counted by the structures that hold pointers to them. - system open file table - process current directory - file system mount points - etc. Each specific file system maintains its own hash of vnodes (BSD). - specific FS handles initialization - free list is maintained by VFS vget(vp): reclaim cached inactive vnode from VFS free list vref(vp): increment reference count on an active vnode vrele(vp): release reference count on a vnode vgone(vp): vnode is no longer valid (file is removed)

Sharing Open File Instances parent user ID process group ID parent PID signal state siblings children child user ID process group ID parent PID signal state siblings children process objects shared seek offset in shared file table entry shared file (inode or vnode) process file descriptors system open file table

File Buffer Cache • Avoid the disk for as many file operations as possible. • Cache acts as a filter for the requests seen by the disk - reads served best. • Delayed writeback will avoid going to disk at all for temp files. Proc Memory File cache

Handling Updates in the File Cache 1. Blocks may be modified in memory once they have been brought into the cache. Modified blocks are dirty and must (eventually) be written back. 2. Once a block is modified in memory, the write back to disk may not be immediate (synchronous). Delayed writes absorb many small updates with one disk write. How long should the system hold dirty data in memory? Asynchronous writes allow overlapping of computation and disk update activity (write-behind). Do the write call for block n+1 while transfer of block n is in progress.

Failures, Commits, Atomicity • What guarantees does the system offer about the hard state if the system fails? – Durability • Did my writes commit, i. e. , are they on the disk? – Atomicity • Can an operation “partly commit”? • Also, can it interleave with other operations? – Recoverability and Corruption • Is the metadata well-formed on recovery?

Unix Failure/Atomicity • File writes are not guaranteed to commit until close. – A process can force commit with a sync. – The system forces commit every (say) 30 seconds. – Failure could lose an arbitrary set of writes. • Reads/writes to a shared file interleave at the granularity of system calls. • Metadata writes are atomic/synchronous. • Disk writes are carefully ordered. – The disk can become corrupt in well-defined ways. – Restore with a scrub (“fsck”) on restart. – Alternatives: logging, shadowing • Want better reliability? Use a database.

The Problem of Disk Layout • The level of indirection in the file block maps allows flexibility in file layout. • “File system design is 99% block allocation. ” [Mc. Voy] • Competing goals for block allocation: – allocation cost – bandwidth for high-volume transfers – stamina/longevity – efficient directory operations • Goal: reduce disk arm movement and seek overhead. • metric of merit: bandwidth utilization Track Sector Arm Head Cylinder Platter

Bandwidth utilization Define b Block size B Raw disk bandwidth (“spindle speed”) s Average access (seek+rotation) delay per block I/O Then Transfer time per block = b/B I/O completion time per block = s + (b/B) Effective disk bandwidth for I/O request stream = b/(s + (b/B)) Bandwidth wasted per I/O: s. B Effective bandwidth utilization (%): b/(s. B + b) How to get better performance? - Larger b (larger blocks, clustering, extents, etc. ) - Smaller s (placement / ordering, sequential access, logging, etc. )

Effective bandwidth (%), B = 40 MB/s

Know your Workload! • File usage patterns should influence design decisions. Do things differently depending: – How large are most files? How long-lived? Read vs. write activity. Shared often? – Different levels “see” a different workload. • Feedback loop Usage patterns observed today File System design and impl

Generalizations from UNIX Workloads • Standard Disclaimers that you can’t generalize…but anyway… • Most files are small (fit into one disk block) although most bytes are transferred from longer files. • Most opens are for read mode, most bytes transferred are by read operations • Accesses tend to be sequential and 100%

More on Access Patterns • There is significant reuse (re-opens) - most opens go to files repeatedly opened & quickly. Directory nodes and executables also exhibit good temporal locality. – Looks good for caching! • Use of temp files is significant part of file system activity in UNIX - very limited reuse, short lifetimes (less than a minute).

Example: BSD FFS • Fast File System (FFS) [Mc. Kusick 81] – Clustering enhancements [Mc. Voy 91], and improved cluster allocation [Mc. Kusick: Smith/Seltzer 96] – FFS can also be extended with metadata logging [e. g. , Episode]

FFS Cylinder Groups • FFS defines cylinder groups as the unit of disk locality, and it factors locality into allocation choices. – typical: thousands of cylinders, dozens of groups – Strategy: place “related” data blocks in the same cylinder group whenever possible. • seek latency is proportional to seek distance – Smear large files across groups: • Place a run of contiguous blocks in each group. – Reserve inode blocks in each cylinder group. • This allows inodes to be allocated close to their directory entries and close to their data blocks (for small files).

FFS Allocation Policies 1. Allocate file inodes close to their containing directories. For mkdir, select a cylinder group with a more-than-average number of free inodes. For creat, place inode in the same group as the parent. 2. Concentrate related file data blocks in cylinder groups. Most files are read and written sequentially. Place initial blocks of a file in the same group as its inode. How should we handle directory blocks? Place adjacent logical blocks in the same cylinder group. Logical block n+1 goes in the same group as block n. Switch to a different group for each indirect block.

Allocating a Block 1. Try to allocate the rotationally optimal physical block after the previous logical block in the file. Skip rotdelay physical blocks between each logical block. (rotdelay is 0 on track-caching disk controllers. ) 2. If not available, find another block a nearby rotational position in the same cylinder group We’ll need a short seek, but we won’t wait for the rotation. If not available, pick any other block in the cylinder group. 3. If the cylinder group is full, or we’re crossing to a new indirect block, go find a new cylinder group. Pick a block at the beginning of a run of free blocks. Provided for completeness

Clustering in FFS • Clustering improves bandwidth utilization for large files read and written sequentially. • Allocate clumps/clusters/runs of blocks contiguously; read/write the entire clump in one operation with at most one seek. – Typical cluster sizes: 32 KB to 128 KB. • FFS can allocate contiguous runs of blocks “most of the time” on disks with sufficient free space. – This (usually) occurs as a side effect of setting rotdelay = 0. • Newer versions may relocate to clusters of contiguous storage if the initial allocation did not succeed in placing them well. – Must modify buffer cache to group buffers together and read/write in contiguous clusters. Provided for completeness

Sequential File Write note sequential block allocation physical disk sector write stall read sync command (typed to shell) pushes indirect blocks to disk sync read next block of free space bitmap (? ? ) time in milliseconds

Sequential Writes: A Closer Look physical disk sector 16 MB in one second (one indirect block worth) 140 ms delay for cylinder seek etc. (? ? ? ) time in milliseconds longer delay for head movement to push indirect blocks write stall

The Problem of Metadata Updates • Metadata updates are a second source of FFS seek overhead. – Metadata writes are poorly localized. • E. g. , extending a file requires writes to the inode, direct and indirect blocks, cylinder group bit maps and summaries, and the file block itself. • Metadata writes can be delayed, but this incurs a higher risk of file system corruption in a crash. – If you lose your metadata, you are dead in the water. – FFS schedules metadata block writes carefully to limit the kinds of inconsistencies that can occur. • Some metadata updates must be synchronous on controllers that don’t respect order of writes.

FFS Failure Recovery FFS uses a two-pronged approach to handling failures: 1. Carefully order metadata updates to ensure that no dangling references can exist on disk after a failure. – Never recycle a resource (block or inode) before zeroing all pointers to it (truncate, unlink, rmdir). – Never point to a structure before it has been initialized. • E. g. , sync inode on creat before filling directory entry, and sync a new block before writing the block map. 2. Run a file system scavenger (fsck) to fix other problems. – Free blocks and inodes that are not referenced. – Fsck will never encounter a dangling reference or double allocation.

Alternative: Logging and Journaling • Logging can be used to localize synchronous metadata writes, and reduce the work that must be done on recovery. • Universally used in database systems. • Used for metadata writes in journaling file systems • Key idea: group each set of related updates into a single log record that can be written to disk atomically (“all-or-nothing”). – Log records are written to the log file or log disk sequentially. • No seeks, and preserves temporal ordering. – Each log record is trailed by a marker (e. g. , checksum) that says “this log record is complete”. – To recover, scan the log and reapply updates.

Metadata Logging Here’s one approach to building a fast filesystem: 1. Start with FFS with clustering. 2. Make all metadata writes asynchronous. But, that approach cannot survive a failure, so: 3. Add a supplementary log for modified metadata. 4. When metadata changes, write new versions immediately to the log, in addition to the asynchronous writes to “home”. 5. If the system crashes, recover by scanning the log. Much faster than scavenging (fsck) for large volumes. 6. If the system does not crash, then discard the log.

Anatomy of a Log head (old). . . Log Sequence Number (LSN) LSN 11 XID 18 LSN 12 XID 18 Transaction ID (XID) commit record LSN 13 XID 19 LSN 14 XID 18 commit force log to stable storage on tail (new) commit • physical • Entries contain item values; restore by reapplying them. • logical (or method logging) • Entries contain operations and their arguments; restore by reexecuting. • redo • Entries can be replayed to restore committed updates (e. g. , new value). • undo • Entries can be replayed to roll back uncommitted updates.

Representing Small Files • Internal fragmentation in the file system blocks can waste significant space for small files. – E. g. , 1 KB files waste 87% of disk space (and bandwidth) in a naive file system with an 8 KB block size. – Most files are small: one study [Irlam 93] shows a median of 22 KB. • FFS solution: optimize small files for space efficiency. – Subdivide blocks into 2/4/8 fragments (or just frags). – Free block maps contain one bit for each fragment. • To determine if a block is free, examine bits for all its fragments. – The last block of a small file is stored on fragment(s). • If multiple fragments they must be contiguous. [Provided for completeness]

Small-File Create Storm 50 MB inodes and file contents (localized allocation) physical disk sector note synchronous writes for some metadata sync delayed-write metadata sync write stall time in milliseconds

Small-File Create: A Closer Look physical disk sector time in milliseconds

Build a Better Disk? • “Better” has typically meant density to disk manufacturers - bigger disks are better. • I/O Bottleneck - a speed disparity caused by processors getting faster more quickly • One idea is to use parallelism of multiple disks – Striping data across disks – Reliability issues - introduce redundancy

RAID Redundant Array of Inexpensive Disks Striped Data (RAID Levels 2 and 3) Parity Disk

Network File System (NFS) server client syscall layer user programs VFS syscall layer NFS server VFS UFS NFS client network

NFS Protocol • NFS is a network protocol layered above TCP/IP. – Original implementations (and most today) use UDP datagram transport for low overhead. • Maximum IP datagram size was increased to match FS block size, to allow send/receive of entire file blocks. • Newer implementations use TCP as a transport. – The NFS protocol is a set of message formats and types for request/response (RPC) messaging.

NFS Vnodes • The NFS protocol has an operation type for (almost) every vnode operation, with similar arguments/results. struct nfsnode* np = VTONFS(vp); syscall layer VFS nfs_vnodeops NFS client stubs nfsnode The nfsnode holds information needed to interact with the server to operate on the file. NFS server RPC/etc. network UFS

Vnode Operations and Attributes vnode attributes (vattr) type (VREG, VDIR, VLNK, etc. ) mode (9+ bits of permissions) nlink (hard link count) owner user ID owner group ID filesystem ID unique file ID file size (bytes and blocks) access time modify time generation number Not to be tested generic operations vop_getattr (vattr) vop_setattr (vattr) vhold() vholdrele() directories only vop_lookup (OUT vpp, name) vop_create (OUT vpp, name, vattr) vop_remove (vp, name) vop_link (vp, name) vop_rename (vp, name, tdvp, tvp, name) vop_mkdir (OUT vpp, name, vattr) vop_rmdir (vp, name) vop_symlink (OUT vpp, name, vattr) vop_readdir (uio, cookie) vop_readlink (uio) files only vop_getpages (page**, count, offset) vop_putpages (page**, count, sync, offset) vop_fsync ()

Pathname Traversal • When a pathname is passed as an argument to a system call, the syscall layer must “convert it to a vnode”. • Pathname traversal is a sequence of vop_lookup calls to descend the tree to the named file or directory. open(“/tmp/zot”) vp = get vnode for / (rootdir) vp->vop_lookup(&cvp, “tmp”); vp = cvp; vp->vop_lookup(&cvp, “zot”); Issues: 1. crossing mount points 2. obtaining root vnode (or current dir) 3. finding resident vnodes in memory 4. caching name->vnode translations 5. symbolic (soft) links 6. disk implementation of directories 7. locking/referencing to handle races with name create and delete operations Provided for completeness

File Handles • Question: how does the client tell the server which file or directory the operation applies to? – Similarly, how does the server return the result of a lookup? • In NFS, the reference is a file handle or fhandle, a token/ticket whose value is determined by the server. – Includes all information needed to identify the file/object on the server, and get a pointer to it quickly. volume ID inode # generation # Typical NFSv 3 Provided for completeness

NFS: Identity/Security • “Classic NFS” was designed for a LAN under common administrative control. – Common uid/gid space – All client kernels are trusted to properly represent the local user identity. – Kernels trusted to control access to cached data. • Volume export (mount privilege) control – Access control list at server – Subjects are nodes (e. g. , DNS name or IP address) • Mount just gives you a root filehandle: those file handles are capabilities.

NFS: From Concept to Implementation • Now that we understand the basics, how do we make it work in a real system? – How do we make it fast? • Answer: caching, read-ahead, and write-behind. – How do we make it reliable? What if a message is dropped? What if the server crashes? • Answer: client retransmits request until it receives a response. – How do we preserve the failure/atomicity model? • Answer: well, we don’t, at least not completely. – What about security and access control?

NFS as a “Stateless” Service The NFS server maintains no transient information about its clients; there is no state other than the file data on disk. Makes failure recovery simple and efficient. • no record of open files • no server-maintained file offsets – Read and write requests must explicitly transmit the byte offset for the operation. • no record of recently processed requests – Retransmitted requests may be executed more than once. – Requests are designed to be idempotent whenever possible. – E. g. , no append mode for writes, and no exclusive create.

File Cache Consistency • Caching is a key technique in distributed systems. – The cache consistency problem: cached data may become stale if cached data is updated elsewhere in the network. • Solutions: – Timestamp invalidation (NFS). • Timestamp each cache entry, and periodically query the server: “has this file changed since time t? ”; invalidate cache if stale. – Callback invalidation (AFS). • Request notification (callback) from the server if the file changes; invalidate cache on callback. – Leases (NQ-NFS) [Gray&Cheriton 89]

Timestamp Validation in NFS [1985] • NFSv 2/v 3 uses a form of timestamp validation like today’s Web – Timestamp cached data at file grain. – Maintain per-file expiration time (TTL) – Probe for new timestamp to revalidate if cache TTL has expired. • Get attributes (getattr) • NFS file cache and access primitives are block-grained, and the client may issue many operations in sequence on the same file. – Clustering: File-grained timestamp for block-grained cache – Piggyback file attributes on each response – Adaptive TTL • What happens on server failure? Client failure?

NQ-NFS Leases • In NQ-NFS, a client obtains a lease on the file that permits the client’s desired read/write activity. • “A lease is a ticket permitting an activity; the lease is valid until some expiration time. ” – A read-caching lease allows the client to cache clean data. • Guarantee: no other client is modifying the file. – A write-caching lease allows the client to buffer modified data for the file. • Guarantee: no other client has the file cached. • Allows delayed writes: client may delay issuing writes to improve write performance (i. e. , client has a writeback cache).

The Synchronous Write Problem • Stateless NFS servers must commit each operation to stable storage before responding to the client. – Interferes with FS optimizations, e. g. , clustering, LFS, and disk write ordering (seek scheduling). • Damages bandwidth and scalability. – Imposes disk access latency for each request. • Not so bad for a logged write; much worse for a complex operation like an FFS file write. • The synchronous update problem occurs for any storage service with reliable update (commit).

Speeding Up NFS Writes • Interesting solutions to the synchronous write problem, used in high-performance NFS servers: • Delay the response until convenient for the server. – E. g. , NFS write-gathering optimizations for clustered writes (similar to group commit in databases). • [NFS V 3 commit operation] – Relies on write-behind from NFS I/O daemons (iods). • Throw hardware at it: non-volatile memory (NVRAM) – Battery-backed RAM or UPS (uninterruptible power supply). – Use as an operation log (Network Appliance WAFL). . . –. . . or as a non-volatile disk write buffer (Legato). • Replicate server and buffer in memory (e. g. , MIT Harp). Provided for completeness

Detecting Server Failure with a Session Verifier “Do A for me. ” “OK, my verifier is x. ” S, x “B” “x” “C” oops. . . “OK, my verifier is y. ” “A and B” S´, y “y” What if y == x? How to guarantee that y != x? What is the implication of re-executing A and B, and after C? Some uses: NFS V 3 write commitment, RPC sessions, NFS V 4 and DAFS (client).

The Retransmission Problem • Sun RPC (and hence NFS) masks network errors by retransmitting each request after a timeout. – Handles dropped requests or dropped replies easily, but an operation may be executed more than once. – Sun RPC has execute-at-least-once semantics, but we need execute-at-most-once semantics for non-idempotent operations. – Retransmissions can radically increase load on a slow server.

Solutions 1. Use TCP or some other transport protocol that produces reliable, in-order delivery. – higher overhead, overkill 2. Implement an execute-at-most once RPC transport. – sequence numbers and timestamps 3. Keep a retransmission cache on the server. – Remember the most recent request IDs and their results, and just resend the result. . does this violate statelessness? 4. Hope for the best and smooth over non-idempotent requests. • Map ENOENT and EEXIST to ESUCCESS.

Recovery in Stateless NFS • If the server fails and restarts, there is no need to rebuild in-memory state on the server. – Client reestablishes contact (e. g. , TCP connection). – Client retransmits pending requests. • Classical NFS uses a connectionless transport (UDP). – Server failure is transparent to the client – No connection to break or reestablish. – Sun/ONC RPC masks network errors by retransmitting a request after an adaptive timeout. • Crashed server is indistinguishable from a slow server. • Dropped packet is indistinguishable from a crashed server.

Drawbacks of a Stateless Service • The stateless nature of classical NFS has compelling design advantages (simplicity), but also some key drawbacks: – Recovery-by-retransmission constrains the server interface. • ONC RPC/UDP has execute-mostly-once semantics (“send and pray”), which compromises performance and correctness. – Update operations are disk-limited. • Updates must commit synchronously at the server. – NFS cannot (quite) preserve local single-copy semantics. • Files may be removed while they are open on the client. • Server cannot help in client cache consistency. • Let’s look at the consistency problem. . .

AFS [1985] • AFS is an alternative to NFS developed at CMU. • Duke still uses it. • Designed for wide area file sharing: – Internet is large and growing exponentially. – Global name hierarchy with local naming contexts and location info embedded in fully qualified names. • Much like DNS – Security features, with per-domain authentication / access control. – Whole file caching or 64 KB chunk caching • Amortize request/transfer cost – Client uses a disk cache • Cache is preserved across client failure. • Again, it looks a lot like the Web. [Provided for completeness]

Callback Invalidations in AFS-2 • AFS-1 uses timestamp validation like NFS; AFS-2 uses callback invalidations. • Server returns “callback promise” token with file access. – Like ownership protocol, confers a right to cache the file. – Client caches the token on its disk. • Token states: {valid, invalid, cancelled} • On a sharing collision, server cancels token with a callback. – Client invalidates cached copy of the associated file. – Detected on client write to server: last writer wins. – (No distinction between read/write token. ) [Provided for completeness]

Issues with Callback Invalidations • What happens after a failure? – Client invalidates its tokens on client restart. • Invalid tokens may be revalidated, like NFS getattr or WWW. – Server must remember tokens across restart. – Can the client distinguish a server failure from a network failure? – Client invalidates tokens after a timeout interval T if the client has no communication with the server. • Weakens consistency in failures. • Then there’s the problem of update semantics: two clients may be actively updating the same file at the same time. [Provided for completeness]

Using NQ-NFS Leases 1. Client NFS piggybacks lease requests for a given file on I/O operation requests (e. g. , read/write). – NQ-NFS leases are implicit and distinct from file locking. 2. The server determines if it can safely grant the request, i. e. , does it conflict with a lease held by another client. – read leases may be granted simultaneously to multiple clients – write leases are granted exclusively to a single client 3. If a conflict exists, the server may send an eviction notice to the holder. – Evicted from a write lease? Write back. – Grace period: server grants extensions while client writes. – Client sends vacated notice when all writes are complete. [Provided for completeness]

NQ-NFS Lease Recovery • Key point: the bounded lease term simplifies recovery. – Before a lease expires, the client must renew the lease. – What if a client fails while holding a lease? • Server waits until the lease expires, then unilaterally reclaims the lease; client forgets all about it. • If a client fails while writing on an eviction, server waits for write slack time before granting conflicting lease. – What if the server fails while there are outstanding leases? • Wait for lease period + clock skew before issuing new leases. – Recovering server must absorb lease renewal requests and/or writes for vacated leases. [Provided for completeness]

NQ-NFS Leases and Cache Consistency – Every lease contains a file version number. • Invalidation cache iff version number has changed. – Clients may disable client caching when there is concurrent write sharing. • no-caching lease (Sprite) – What consistency guarantees do NQ-NFS leases provide? • Does the server eventually receive/accept all writes? • Does the server accept the writes in order? • Are groups of related writes atomic? • How are write errors reported? • What is the relationship to NFS V 3 commit? [Provided for completeness]

Using Disk Storage • Typical operating systems use disks in three different ways: • System calls allow user programs to access a “raw” disk. – Unix: special device file identifies volume directly. – Any process that can open the device file can read or write any specific sector in the disk volume. • OS uses disk as backing storage for virtual memory. – OS manages volume transparently as an “overflow area” for VM contents that do not “fit” in physical memory. • 3. OS provides syscalls to create/access files residing on disk. – OS file system modules virtualize physical disk storage as a collection of logical files. [Provided for completeness]

Alternative Structure: DOS FAT Disk Blocks 0 EOF 13 snow: 6 rain: 5 hail: 10 directory 2 9 FAT 8 root directory FREE 4 12 3 FREE EOF FREE [Provided for completeness]