Google Bigtable Fay Chang Jeffrey Dean Sanjay Ghemawat

Google Bigtable Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, Robert E. Gruber Google, Inc. OSDI 2006 Adapted by S. Sudarshan from a talk by Erik Paulson, UW Madison

Google Scale n Lots of data ¨ Copies of the web, satellite data, user data, email and USENET, Subversion backing store n n Many incoming requests No commercial system big enough ¨ Couldn’t afford it if there was one ¨ Might not have made appropriate design choices n n Firm believers in the End-to-End argument 450, 000 machines (NYTimes estimate, June 14 th 2006 2

Building Blocks Scheduler (Google Work. Queue) n Google Filesystem n Chubby Lock service n Two other pieces helpful but not required n ¨ Sawzall ¨ Map. Reduce n (despite what the Internet says) Big. Table: build a more application-friendly storage service using these parts 3

Google File System Large-scale distributed “filesystem” n Master: responsible for metadata n Chunk servers: responsible for reading and writing large chunks of data n Chunks replicated on 3 machines, master responsible for ensuring replicas exist n OSDI ’ 04 Paper n 4

Chubby {lock/file/name} service n Coarse-grained locks, can store small amount of data in a lock n 5 replicas, need a majority vote to be active n Also an OSDI ’ 06 Paper n 5

Data model: a big map n n <Row, Column, Timestamp> triple for key - lookup, insert, and delete API Arbitrary “columns” on a row-by-row basis Column family: qualifier. Family is heavyweight, qualifier lightweight ¨ Column-oriented physical store- rows are sparse! ¨ n Does not support a relational model No table-wide integrity constraints ¨ No multirow transactions ¨ 6

SSTable Immutable, sorted file of key-value pairs n Chunks of data plus an index n ¨ Index is of block ranges, not values 64 K block SSTable Index 7

Tablet Contains some range of rows of the table n Built out of multiple SSTables n Tablet 64 K block Start: aardvark 64 K block End: apple SSTable Index 64 K block SSTable Index 8

Table n n n Multiple tablets make up the table SSTables can be shared Tablets do not overlap, SSTables can overlap Tablet aardvark Tablet apple SSTable apple_two_E boat SSTable 9

Finding a tablet n n Stores: Key: table id + end row, Data: location Cached at clients, which may detect data to be incorrect ¨ n in which case, lookup on hierarchy performed Also prefetched (for range queries) 10

Servers n Tablet servers manage tablets, multiple tablets per server. Each tablet is 100 -200 MB ¨ Each tablet lives at only one server ¨ Tablet server splits tablets that get too big n Master responsible for load balancing and fault tolerance 11

Master’s Tasks n Use Chubby to monitor health of tablet servers, restart failed servers ¨ Tablet server registers itself by getting a lock in a specific directory chubby n n Chubby gives “lease” on lock, must be renewed periodically Server loses lock if it gets disconnected ¨ Master monitors this directory to find which servers exist/are alive n n If server not contactable/has lost lock, master grabs lock and reassigns tablets GFS replicates data. Prefer to start tablet server on same machine that the data is already at 12

Master’s Tasks (Cont) n When (new) master starts ¨ grabs n master lock on chubby Ensures only one master at a time ¨ Finds live servers (scan chubby directory) ¨ Communicates with servers to find assigned tablets ¨ Scans metadata table to find all tablets Keeps track of unassigned tablets, assigns them n Metadata root from chubby, other metadata tablets assigned before scanning. n 13

Metadata Management n Master handles ¨ table n creation, and merging of tablet Tablet servers directly update metadata on tablet split, then notify master ¨ lost notification may be detected lazily by master 14

Editing a table Mutations are logged, then applied to an in-memory memtable May contain “deletion” entries to handle updates ¨ Group commit on log: collect multiple updates before log flush ¨ Tablet Memtable Insert tablet log n Delete Memory apple_two_E boat Insert Delete Insert SSTable GFS 15

Compactions n Minor compaction – convert the memtable into an SSTable ¨ Reduce memory usage ¨ Reduce log traffic on restart n Merging compaction ¨ Reduce number of SSTables ¨ Good place to apply policy “keep only N versions” n Major compaction ¨ Merging compaction that results in only one SSTable ¨ No deletion records, only live data 16

Locality Groups n Group column families together into an SSTable Avoid mingling data, e. g. page contents and page metadata ¨ Can keep some groups all in memory ¨ n n Can compress locality groups Bloom Filters on SSTables in a locality group bitmap on keyvalue hash, used to overestimate which records exist ¨ avoid searching SSTable if bit not set ¨ n Tablet movement Major compaction (with concurrent updates) ¨ Minor compaction (to catch up with updates) without any concurrent updates ¨ Load on new server without requiring any recovery action ¨ 17

Log Handling n Commit log is per server, not per tablet (why? ) ¨ complicates tablet movement ¨ when server fails, tablets divided among multiple servers n n n cause heavy scan load by each such server optimization to avoid multiple separate scans: sort log by (table, rowname, LSN), so logs for a tablet are clustered, then distribute GFS delay spikes can mess up log write (time critical) ¨ solution: two separate logs, one active at a time ¨ can have duplicates between these two 18

Immutability n SSTables are immutable ¨ simplifies caching, sharing across GFS etc ¨ no need for concurrency control ¨ SSTables of a tablet recorded in METADATA table ¨ Garbage collection of SSTables done by master ¨ On tablet split, split tables can start off quickly on shared SSTables, splitting them lazily n Only memtable has reads and updates concurrent ¨ copy on write rows, allow concurrent read/write 19

Microbenchmarks 20

21

Application at Google 22

Lessons learned Interesting point- only implement some of the requirements, since the last is probably not needed n Many types of failure possible n Big systems need proper systems-level monitoring n Value simple designs n 23