CPS 110 Introduction to Google Landon Cox April

CPS 110: Introduction to Google Landon Cox April 10, 2008

A brief history of Google = Back. Rub: 1996 4 disk drives 24 GB total storage

A brief history of Google = Google: 1998 44 disk drives 366 GB total storage

A brief history of Google: 2003 15, 000 machines ? PB total storage

Google design principles ê Workload: easy to parallelize ê Want to take advantage of many processors, disks ê Why not buy a bunch of supercomputers? ê Leverage parallelism of lots of (slower) cheap machines ê Supercomputer price/performance ratio is poor ê What is the downside of cheap hardware? ê Lots of hardware failures 1. Use lots of cheap, commodity hardware 2. Provide reliability in software

What happens on a query? http: //www. google. com/search? q=duke http: //64. 233. 179. 104/search? q=duke DNS

What happens on a query? http: //64. 233. 179. 104/search? q=duke Spell Checker Ad Server Index Servers (TB) Document Servers (TB)

Google hardware model ê Google machines are cheap and likely to fail ê What must they do to keep things up and running? ê Store data in several places (replication) ê When one machine fails, shift load onto ones still around ê Does replication get you anything else? ê Enables more parallel reads ê Better performance ê Good since vast majority of Google traffic is reads

Fault tolerance and performance ê Google machines are cheap and likely to fail ê Does it matter how fast an individual machine is? ê Somewhat, but not that much ê Parallelism enabled by replication has a bigger impact ê Any downside to having a ton of machines? ê Space ê Power consumption ê Cooling costs

Fault tolerance and performance ê Google machines are cheap and likely to fail ê Any workloads where this wouldn’t work? ê Lots of writes to the same data ê Web examples? (web is mostly read)

Google power consumption ê A circa 2003 mid-range server ê Draws 90 W of DC power under load ê 55 W for two CPUs ê 10 W for disk drive ê 25 W for DRAM and motherboard ê Assume 75% efficient ATX power supply ê 120 W of AC power per server ê 10 k. W per rack

Google power consumption ê A server rack fits comfortably in 25 ft 2 ê Power density of 400 W/ ft 2 ê Higher-end server density = 700 W/ ft 2 ê Typical data centers provide 70 -150 W/ ft 2 ê Google needs to bring down the power density ê Requires extra cooling or space ê Lower power servers? ê Slower, but must not harm performance ê Depreciate faster, but must not affect price/performance

Course administration ê Project 3 ê Spec is out ê Read “Smashing the stack” ê Next week ê No class on Thursday ê Discussion section Tuesday, Wednesday

Google storage ê “The Google File System” ê Award paper at SOSP in 2003 ê You should read the paper ê Course review masquerading as new material ê Interesting read ê If you enjoy reading the paper ê Sign up for CPS 210 (you’ll read lots of papers like it!)

Google design principles 1. Use lots of cheap, commodity hardware 2. Provide reliability in software 4 Scale ensures a constant stream of failures 4 2003: > 15, 000 machines 4 2006: > 100, 000 machines 4 GFS exemplifies how they manage failure

Sources of failure ê Software ê Application bugs, OS bugs ê Human errors ê Hardware ê Disks, memory ê Connectors, networking ê Power supplies

Design considerations 1. Component failures 2. Files are huge (multi-GB files) ê Recall that PC files are mostly small ê How did this influence PC FS design? ê Relatively small block size (~KB)

Design considerations 1. Component failures 2. Files are huge (multi-GB files) 3. Most writes are large, sequential appends ê Old data is rarely over-written

Design considerations 1. 2. 3. 4. Component failures Files are huge (multi-GB files) Most writes are large, sequential appends Reads are large and streamed or small and random ê Once written, files are only read, often sequentially ê Is this like or unlike PC file systems? ê PC reads are mostly sequential reads of small files ê How do sequential reads of large files affect client caching? ê Caching is pretty much useless

Design considerations 1. 2. 3. 4. Component failures Files are huge (multi-GB files) Most writes are large, sequential appends Reads are large and streamed or small and random 5. Design file system for apps that use it ê Files are often used as producer-consumer queues ê 100 s of producers trying to append concurrently ê Want atomicity of append with minimal synchronization ê Want support for atomic append

Design considerations 1. 2. 3. 4. Component failures Files are huge (multi-GB files) Most writes are large, sequential appends Reads are large and streamed or small and random 5. Design file system for apps that use it 6. High sustained bandwidth better than low latency ê What is the difference between BW and lantency? ê Network as road (BW = # lanes, latency = speed limit)

Google File System (GFS) ê Similar API to POSIX ê Create/delete, open/close, read/write ê GFS-specific calls ê Snapshot (low-cost copy) ê Record_append ê (allows concurrent appends, ensures atomicity of each append) ê What does this description of record_append mean? ê Individual appends may be interleaved arbitrarily ê Each append’s data will not be interleaved with another’s

GFS architecture ê Cluster-based ê Single logical master ê Multiple chunkservers ê Clusters are accessed by multiple clients ê Clients are commodity Linux machines ê Machines can be both clients and

GFS architecture

File data storage ê Files are broken into fixed-size chunks ê Chunks are named by a globally unique ID ê ID is chosen by the master ê ID is called a chunk handle ê Servers store chunks as normal Linux files ê Servers accept reads/writes with handle + byte range

File data storage ê Chunks are replicated at 3 servers ê What are the advantages of replication? ê Better availability (if one fails, two left) ê Better read performance (parallel reads)

File data storage ê Chunks are replicated at 3 servers ê Servers lease right to serve a chunk ê Responsible for a chunk for a period of time ê Must renew lease when it expires ê How does this make failure easier? ê If a node fails, its leases will expire ê When it comes back up, just renew leases

File meta-data storage ê Master maintains all meta-data ê What do we mean by meta-data? ê Namespace info ê Access control info ê Mapping from files to chunks ê Current chunk locations

Other master responsibilities ê Chunk lease management ê Garbage collection of orphaned chunks ê How might a chunk become orphaned? ê If a chunk is no longer in any file ê Chunk migration between servers ê Heart. Beat messages to chunkservers

Client details ê Client code is just a library ê Similar to File class in java ê Caching ê No in-memory data caching at the client or servers ê Clients still cache meta-data ê Why don’t servers cache data (trick question)? ê Chunks are stored as regular Linux files ê Linux’s in-kernel buffer cache already caches chunk content

Master design issues ê Single (logical) master per cluster ê Master’s state is actually replicated elsewhere ê Logically single because client speaks to one name ê Use DNS tricks to locate/talk to a master ê Pros ê Simplifies design ê Master endowed with global knowledge ê (makes good placement, replication decisions)

Master design issues ê Single (logical) master per cluster ê Master’s state is actually replicated elsewhere ê Logically single because client speak to one name ê Cons? ê Could become a bottleneck ê (recall how replication can improve performance) ê How to keep from becoming a bottleneck? ê Minimize its involvement in reads/writes ê Clients talk to master very briefly ê Most communication is with chunkservers

Example read ê Client uses fixed size chunks to compute chunk index within a file ê Does this remind you of anything? ê Computing virtual page numbers in Project 2

Example read ê Client asks master for the chunk handle at index i of the file

Example read ê Master replies with the chunk handle and list of replicas

Example read ê Client caches handle and replica list ê (maps filename + chunk index chunk handle + replica list) ê (kind of like your project 2’s per-virtual page state)

Example read ê Client sends a request to the closest chunk server ê Server returns data to client

Example read ê Any possible optimizations? ê Could ask for multiple chunk handles at once (batching) ê Server could return handles for sequent indices (pre-fetching)

Chunk size ê Recall how we chose block/page sizes ê What are the disadvantages of small/big chunks? 1. If too small, too much storage used for meta-data 2. If too large, too much internal fragmentation ê Impact of chunk size on client caching? ê Data chunks are not cached (so no impact there) ê Large chunks less meta-data/chunk ê Clients can cache more meta-data at clients

Chunk size ê Recall how we chose block/page sizes ê What are the disadvantages of small/big chunks? 1. If too small, too much storage used for meta-data 2. If too large, too much internal fragmentation ê Impact of chunk size on master performance? ê Large chunks less meta-data/chunk ê Masters can fit all meta-data in memory ê Much faster than retrieving from disk

Chunk size ê Recall how we chose block/page sizes ê What are the disadvantages of small/big chunks? 1. If too small, too much storage used for meta-data 2. If too large, too much internal fragmentation ê What is a reasonable chunk size then? ê They chose 64 MB ê Reasonable when most files are many GB

Master’s meta-data 1. File and chunk namespaces 2. Mapping from files to chunks 3. Chunk replica locations ê All are kept in-memory ê 1. and 2. are kept persistent ê Use an operation log

Operation log ê Historical record of all meta-data updates ê Only persistent record of meta-data updates ê Replicated at multiple machines ê Appending to log is transactional ê Log records are synchronously flushed at all replicas ê To recover, the master replays the operation log

Atomic record_append ê Traditional write ê Concurrent writes to same file region are not serialized ê Region can end up containing fragments from many clients ê Record_append ê Client only specifies the data to append ê GFS appends it to the file at least once atomically ê GFS chooses the offset ê Why is this simpler than forcing clients to synchronize? ê Clients would need a distributed locking scheme ê GFS provides an abstraction, hides concurrency issues from clients

Snapshots in GFS ê Allows clients to quickly copy subtrees ê How do you make copying fast? ê Copy-on-write ê Point new meta-data to old data ê If old data is overwritten, make a copy ê Defers the work of copying

Snapshots in GFS ê Cannot snapshot concurrently with writes ê What does master do before snapshot? ê Waits for all leases to expire ê No writes if no outstanding leases ê After all leases revoked/expired ê Master makes meta-data for new snapshot

Snapshots in GFS ê After all leases revoked/expired ê Master makes meta-data for new snapshot ê What happens if client writes to snapshot? ê Leases expired so request goes to Master ê Master tells each server to make copy of chunk ê Nice because copying is local, not over network

GFS namespace ê Looks like a tree but isn’t really ê No per-directory data that lists files in directory ê Strictly a mapping from names to metadata ê Idea is to increase concurrency

Namespace and locking ê All namespace ops execute at Master ê Could put a big lock on namespace state ê Why is this a bad idea? ê Very little concurrency ê What should be done instead? ê Assign a lock to each name

Example operation ê Two concurrent operations ê Create /home/user/foo ê Snapshot /home/user /save/user ê Don’t want these operations to happen at same time ê What locks does snapshot operation acquire? ê Read locks for /home and /save ê Write locks for /home/user and /save/user ê What locks does create operation acquire? ê Read locks for /home and /home/user ê Write locks for /home/user/foo

Example operation ê Two concurrent operations ê Create /home/user/foo ê Snapshot /home/user /save/user ê Don’t want these operations to happen at same time ê Why read lock on /home/user ? ê Sufficient to prevent parent from being deleted ê Allows parallel manipulations of “tree” ê Why write lock on /home/user/foo ? ê Sufficient to serialize concurrent creates/deletes of file

Mutation order ê Mutations are performed at each chunk’s replica ê Master chooses a primary for each chunk ê Others are called secondary replicas ê Primary chooses an order for all mutations ê Called “serializing” ê All replicas follow this “serial” order

Example mutation ê Client asks master ê Primary replica ê Secondary replicas

Example mutation ê Master returns ê Primary replica ê Secondary replicas

Example mutation ê Client sends data ê To all replicas ê Replicas ê Only buffer data ê Do not apply ê Ack client

Example mutation ê Client tells primary ê Write request ê Identifies sent data ê Primary replica ê Assigns serial #s ê Writes data locally ê (in serial order)

Example mutation ê Primary replica ê Forwards request ê to secondaries ê Secondary replicas ê Write data locally ê (in serial order)

Example mutation ê Secondary replicas ê Ack primary ê Like “votes”

Example mutation ê Primary replica ê Ack client ê Like a commit

Example mutation ê Errors? ê Require consensus ê Just retry

Upcoming classes ê Exam review ê Course evaluations ê More fun with Google