File and Storage Systems The Google File System

File and Storage Systems: The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google Presented by – Uma Murthy CS 5204 Fall 2004 1

Key Ideas (1) n A new file system for Google to satisfy the demands of its application workloads and technological environment CS 5204 Fall 2004 2

Key Ideas (2) n Scalability q n Fast Recovery q n Separate data flow from control flow Availability q n Metadata High aggregate throughput q n Cluster architecture Replication and a relaxed consistency model Fault tolerance q Replication and data integrity CS 5204 Fall 2004 3

Technological Environment n n n Clusters of few thousand commodity Linux machines (versus a small number of highend servers) Multiple clusters are distributed at different geographical locations Reliability is provided at software level by replicating services and automatically detecting and handling failures CS 5204 Fall 2004 4

Google Tasks n n Crawling Searching n Data Stored n n Web documents Indices Spell Checker Google Web Server Ad Server Document Servers Index Servers Serving a Google Query CS 5204 Fall 2004 5

A New File System for Google n Component failures norm rather than exception q n File size q q n Multi-GB files Revisit I/O operation and block size Workloads q q n Clusters of inexpensive commodity machines Reads – Sequential and Random Writes – Dominated by appending new data rather than overwriting existing data Benefits of increasing flexibility as applications and file system API are co-designed q Relaxed consistency model CS 5204 Fall 2004 6

Architecture (file name, chunk index) Application GFS Master File Namespace /foo/bar chunk 2 ef 0 (chunk handle, chunk locations) GFS Client Instructions to chunkserver Chunkserver state (chunk handle, byte range) chunk data GFS chunkserver Linux file system Data messages Control messages ……. Chunks CS 5204 Fall 2004 7

Metadata n Master q q File and chunk namespaces Mapping from files to chunks Location of each chunk’s replicas (not persistent) Operation Log n n Serves as logical timeline that defines the order of concurrent operations Replicated on multiple remote machines Master checkpoints its state when log grows beyond a certain size Chunkserver q q Checksums for each 64 KB block of user data Chunk version number CS 5204 Fall 2004 8

Leases and Mutation Order n Use of leases to maintain consistent mutation order q n n A lease, when granted, has a timeout that may be extended Master grants chunk lease to primary Global mutation order defined by q q The lease grant order chosen by the master Within a lease by the serial numbers assigned by the primary CS 5204 Fall 2004 9

Write Control and Data Flow 4 1 Client Master 2 1. Request for lease information 2. Reply with identity of primary and secondaries 3. Push data to all replicas 4. Send write request to primary 5. Primary forwards write request to all secondaries 6. Secondaries reply after completing operation 7. Primary replies to client. Error handling 3 Secondary Replica A 6 7 Primary Replica 5 Data Secondary Replica B 6 Control CS 5204 Fall 2004 10

Replica Placement n n Multi-level distribution of replicas Chunk replica placement policy q q n Maximizes data reliability and availability Maximizes network bandwidth utilization Place replicas across different machines across different racks CS 5204 Fall 2004 11

Creation, Re-replication, Rebalancing n n Master creates a chunk and chooses placement of initial replicas Re-replication takes place as soon as the number of replicas falls below user specified level q n Master picks highest priority chunk and “clones” it Master rebalances replicas for better disk space and load balancing CS 5204 Fall 2004 12

Garbage Collection n Storage is reclaimed lazily for deleted files at file and chunk levels q q q n Deleted files are first renamed to a hidden name with timestamp Master scans for and removes any “hidden” files that have existed for more than a configurable period In-memory metadata is erased for removed files. Orphaned chunks q q Identified during scan of chunk namespace and metadata is erased for those chunks Chunkservers are notified via Heart. Beat messages and are free to delete replicas CS 5204 Fall 2004 13

Stale Replica Deletion n Chunk replicas become stale if a chunkserver fails and misses mutations to the chunk Chunk version number is used to distinguish between up-to-date and stale replicas Stale replicas are removed during master’s regular garbage collection CS 5204 Fall 2004 14

Master Replication n n Master state, its operation logs and checkpoints are replicated on many machines Since metadata is maintained in-memory, when master fails, it can restart instantly When master machine or disk fails, monitoring infrastructure outside GFS starts a new master process elsewhere Shadow masters provide read-only access to the file system when the primary master is down CS 5204 Fall 2004 15

Microbenchmarks • Aggregate read rate is 75% of theoretical limit • Aggregate write rate is 50% of theoretical limit • For record appends, performance is limited by network bandwidth of chunkservers that store the last chunk of the file independent of the number of clients CS 5204 Fall 2004 16

Real World Clusters (1) n Storage q n File data replicated thrice Metadata q q Chunkservers – checksums, chunk version number Master - 100 bytes per file in master CS 5204 Fall 2004 17

Real World Clusters (2) n Read and Writes q n Read rates much higher than write rates Master Load q 200 -500 operations/sec • Recovery Time – 150000 chunks containing 600 GB data restored in 23. 2 minutes, effective replication rate of 440 MB/s. – Replication of 266 chunks in 2 minutes CS 5204 Fall 2004 18

Evaluation (1) n n A new file system with an effective design built using old techniques The file system is specific to Google applications but some ideas may be applied to data processing tasks of similar size The whole file system is dependent on a single master as the global view of the system is present only in the master. If the master goes down, entire system comes to a halt for some time Not many details on actual size of the file system leads to speculation CS 5204 Fall 2004 19

Evaluation (2) Relation to other papers read n n Follows the BASE data semantics Importance of metadata and metadata update operation. Plays a key role in recovery q Uses journaling metadata update technique CS 5204 Fall 2004 20

Questions? CS 5204 Fall 2004 21