Distributed File Systems Sarah Diesburg Operating Systems CS

Distributed File Systems Sarah Diesburg Operating Systems CS 3430

Distributed File System l Provides transparent access to files stored on a remote disk l Recurrent themes of design issues ¡Failure handling ¡Performance optimizations ¡Cache consistency

No Client Caching l Use RPC to forward every file system request to the remote server ¡open, seek, read, write Server cache: X read Client A write Client B

No Client Caching + Server always has a consistent view of the file system - Poor performance - Server is a single point of failure

Network File System (NFS) l Uses client caching to reduce network load l Built on top of RPC Server cache: X Client A cache: X Client B cache: X

Network File System (NFS) + Performance better than no caching - Has to handle failures - Has to handle consistency

Failure Modes l If the server crashes ¡Uncommitted data in memory are lost ¡Current file positions may be lost ¡The client may ask the server to perform unacknowledged operations again l If a client crashes ¡Modified data in the client cache may be lost

NFS Failure Handling 1. Write-through caching 2. Stateless protocol: the server keeps no state about the client ¡read open, seek, read, close ¡No server recovery after a failure 3. Idempotent operations: repeated operations get the same result ¡No static variables

NFS Failure Handling 4. Transparent failures to clients ¡Two options l. The client waits until the server comes back l. The client can return an error to the user application • Do you check the return value of close?

NFS Weak Consistency Protocol l A write updates the server immediately l Other clients poll the server periodically for changes l No guarantees for multiple writers

NFS Summary + Simple and highly portable - May become inconsistent sometimes ¡Does not happen very often

Andrew File System (AFS) l Developed at CMU l Design principles ¡Files are cached on each client’s disks l. NFS caches only in clients’ memory ¡Callbacks: The server records who has the copy of a file ¡Write-back cache on file close. The server then tells all clients that own an old copy. ¡Session semantics: Updates are only visible on close

AFS Illustrated Server cache: X Client A Client B

AFS Illustrated Server cache: X callback list of X client A read X Client B

AFS Illustrated Server cache: X callback list of X client A read X Client A cache: X read X Client B

AFS Illustrated Server cache: X callback list of X client A read X Client A cache: X Client B read X

AFS Illustrated Server cache: X callback list of X client A client B read X Client A cache: X Client B read X

AFS Illustrated Server cache: X callback list of X client A client B read X Client A cache: X Client B cache: X read X

AFS Illustrated Server cache: X Client A cache: X write X, X X Client B cache: X

AFS Illustrated Server cache: X X X Client A cache: X close X Client B cache: X

AFS Illustrated Server cache: X X X Client A cache: X close X Client B cache: X

AFS Illustrated Server cache: X Client A cache: X Client B cache: X close X

AFS Illustrated Server cache: X X Client A cache: X Client B cache: X open X

AFS Illustrated Server cache: X X Client A cache: X Client B cache: X open X

AFS Failure Handling l If the server crashes, it asks all clients to reconstruct the callback states

AFS vs. NFS l AFS ¡Less server load due to clients’ disk caches ¡Not involved for read-only files l Both AFS and NFS ¡Server is a performance bottleneck ¡Single point of failure

Serverless Network File Service (x. FS) l Idea: construct a file system as a parallel program and exploit the high-speed LAN ¡Four major pieces l. Cooperative caching l. Write-ownership cache coherence l. Software RAID l. Distributed control

Cooperative Caching l Uses remote memory to avoid going to disk ¡On a cache miss, check the local memory and remote memory, before checking the disk ¡Before discarding the last cached memory copy, send the content to remote memory if possible

Cooperative Caching Client A cache: X Client B cache: Client C cache: Client D cache:

Cooperative Caching Client A cache: X Client B cache: X Client C cache: read X Client D cache:

Cooperative Caching Client A cache: X Client B cache: X Client C cache: X read X Client D cache:

Write-Ownership Cache Coherence l Declares a client to be a owner of the file at writes ¡No one else can have a copy

Write-Ownership Cache Coherence owner, read-write Client A cache: X Client B cache: Client C cache: Client D cache:

Write-Ownership Cache Coherence owner, read-write Client A cache: X Client B cache: Client C cache: Client D cache: read X

Write-Ownership Cache Coherence read-only Client A cache: X Client B cache: X Client C cache: read X Client D cache:

Write-Ownership Cache Coherence read-only Client A cache: X Client B cache: X Client C cache: X read-only Client D cache:

Write-Ownership Cache Coherence read-only Client A cache: X Client B cache: Client C cache: X Client D cache: read-only write X

Write-Ownership Cache Coherence Client A cache: Client B cache: Client C cache: X Client D cache: owner, read-write X

Other components l Software RAID ¡Stripe data redundantly over multiple disks l Distributed control ¡File system managers are spread across all machines

x. FS Summary l Built on small, unreliable components l Data, metadata, and control can live on any machine l If one machine goes down, everything else continues to work l When machines are added, x. FS starts to use their resources

x. FS Summary - Complexity and associated performance degradation - Hard to upgrade software while keeping everything running