Case studies of largescale systems Dynamo Amazon Big

Case studies of large-scale systems Dynamo (Amazon) Big. Table (Google) Jinyang Li (with slides adapted from Dynamo and Big. Table authors)

Whole system design • So far, we have learnt various tools – Logging, Replication, 2 P commit, RSM, Paxos, Versions, Content hashes… • Whole system design requires more than a collection of tools • Key question: what is the right abstraction? – Need to understand applications’ usage – Need to make the right tradeoffs

Whole system design • Some design is heavily influenced by the need to re-use existing infrastructures – Re-use mature software saves development efforts – E. g. Big. Table uses Chubby, GFS, SSTable, Sawzall … – Modular, composable design maximizes re-use • Simplicity is key to success – Be a minimalist: avoid adding features unless you must

Whole system design • Today: two example system design • Dynamo: a key-value store • Big. Table: a structured storage system • What are their abstractions? • What are the design tradeoffs? • Why the particular design? Can it be simpler?

Case study I: Dynamo: Amazon’s Highly Available Key-value Store

Dynamo’s goal • Build a distributed storage system: – – Scale Simple: key-value Highly available Guarantee Service Level Agreements (SLA) • Must satisfy strict performance bounds • E. g. 300 ms for 99. 9% requests for a peak load of 500 req/s

System Assumptions and Requirements • Query Model: simple read and write operations to a data item that is uniquely identified by a key. • Consistency requirements: Eventual Consistency (tradeoff high availability for strong consistency) • Efficiency: latency requirements which are in general measured at the 99. 9 th percentile of the distribution. • Other Assumptions: no security related requirements such as authentication and authorization.

Design tradeoffs • Sacrifice strong consistency for availability • Conflict resolution is executed during read instead of write, i. e. “always writeable”. • Other desired properties: – – Incremental scalability. Symmetry. Decentralization. Heterogeneity.

Techniques used in Dynamo Problem Technique Partition data among nodes Consistent Hashing Highly available writes Eventual consistency using vector clocks for reconciliation during reads Handle temporary failures Sloppy quorum and hinted handoff Recover from permanent failures Anti-entropy using Merkle trees Membership and failure detection • Manual addition/deletion of nodes • Gossip current node status

Partition data among nodes • Use consistent hashing • Each node is responsible for log(n) virtual nodes – More load balanced – More capable nodes use more virtual nodes

Data replication • Each data item is replicated at N hosts. • Preference list contains >N successors eligible for storing a particular key.

Data Versioning • A put may return to its caller before the update has been applied at all the replicas • A get may return many versions of the same object. • Challenge: reconcile conflicting versions of the same objects. • Solution: vector clocks

Vector Clock (i. e. version vector) • A vector clock is a list of (node, counter) pairs. • Every version of every object is associated with one vector clock. • VVx ≤ Vvy iff every component of VVx ≤ every component of VVy

Vector clock example

Sloppy Quorum • R, W is the min # of nodes that must participate in a successful read/write op. • Apps tune performance/consistency tradeoffs by adjusting R, W – E. g. R + W > N yields a quorum-like system. – A smaller R (or W) results in better latency

Other techniques • Hinted handoff: – Write to the N+1 -th successor (B) if some of the N immediate successors (A) is unavailable – B hands off data to A where A comes back up • Replica synchronization: – Merkle hash tree. • Membership and Failure Detection: – Manual addition/deletion – Gossip-based status update

Implementation • What existing software does Dynamo use? – Berkeley Database (BDB) Transactional Data Store: for objects of 10 s KB – My. SQL for objects >10 s KB – BDB Java Edition, etc.

Evaluation

Case study II: Google’s Big. Table

Google’s problem: Crawling the Web • Large volumes of data – Over 10 billion URLs – Page contents, link graph, pagerank, etc. • Variety in size • Variety in usage patterns – Random-access reads/writes – Bulk reads/writes • Design goals – Scale to thousands of servers, petabyte of data 3/2/2021 Google, Inc. 21

Big. Table design goals • Scalability – – Terabytes served from memory Petabyte served from disk Thousands of servers Millions of reads/writes per second, efficient bulk reads • Self-managing – Servers adjust to load – Incremental deployment of computing resources 3/2/2021 Google, Inc. 22

Data Model (i. e. abstraction) • Multi-dimensional sparse sorted map (row, column, timestamp) value “contents: ” “cnn. com” “<html>…” 12/9/05 -11: 00 5/4/06 -12: 14: 02 10/31/06 -05: 40: 36 • This abstraction is chosen to match Google’s applications 3/2/2021 Google, Inc. 23

Data Model • Row keys (string) – Primary index: userid, URL – Primary decomposition (data is partitioned among nodes by rows) • Column keys (string) – Field selector – Secondary decomposition (data is grouped for storage by columns) • Timestamp (microsecond granularity) – Useful for versioned data – No interpretation except for order 3/2/2021 Google, Inc. 24

Tablets • A Tablet is a dynamic partition of rows – Distributed over servers – Tablets split and merge automatically based on size and load – Clients can choose row keys to achieve locality • Ownership – A given tablet is owned by one server – One server owns many tablets – Load balancing moves tablet ownership 3/2/2021 Google, Inc. 25

Tablets “contents: ” “language: ” “pagerank: ” “aaa. com” “cnn. com” “<html>…” EN “cnn. com/sports. html” 0. 65 server 1 … “website. com” “yahoo. com/kids. html” server 3 server 2 “zuppa. com/menu. html” server 2 3/2/2021 Google, Inc. 26

Locality Groups • Dynamic partition of columns – Segregates data within a tablet – Different locality groups => different files on GFS • Provides control over storage layout – Memory mapping of locality groups – Choice of compression algorithms – Client-controlled block size 3/2/2021 Google, Inc. 27

Locality Groups “contents: ” “language: ” “pagerank: ” “aaa. com” “cnn. com” “<html>…” … EN 0. 65 “cnn. com/sports. html” … “website. com” “yahoo. com/kids. html” “zuppa. com/menu. html” 3/2/2021 Google, Inc. 28

Other Features • Access control • Qo. S/Prioritization� • Cross-cell replication – Eventual consistency • Chained writes • Integration with Map. Reduce [OSDI ‘ 04] 3/2/2021 Google, Inc. 29

Google’s Building Blocks • Google File System [SOSP ‘ 03] – Raw storage, distributed file system – Stores data and commit logs • SSTable • Chubby [OSDI ‘ 06] – Distributed lock service/small-file system – Performs master election, location bootstrapping – Stores metadata • Cluster scheduling system • RPC system • Authentication system 3/2/2021 Google, Inc. 30

Masters C 0 C 1 C 5 C 2 Chunkserver 1 • • Replicas Google File System (GFS) Client GFS Master C 1 C 5 Misc. servers GFS Master C 3 Chunkserver 2 Client C 0 … C 5 C 2 Chunkserver N Master manages metadata Data transfers happen directly between clients/chunkservers Files broken into chunks (typically 64 MB) Chunks replicated across three machines 3/2/2021 Google, Inc. 31

Typical Google Cluster Machine 1 User app 2 Scheduler slave Bigtable master GFS chunkserver Scheduler slave Cluster scheduling master User app 2 Tablet server User app 1 Linux 3/2/2021 Machine N Machine 2 GFS chunkserver … Tablet server Scheduler slave GFS chunkserver Linux Chubby Google, Inc. GFS master 32

Bigtable System Structure table creation, ACL manipulation, etc. Bigtable client library Bigtable master Read()/Write() Bigtablet server GFS 3/2/2021 … Open() Bigtablet server Chubby Google, Inc. 33

Location Tree Location table Actual tablet in table T Root tablet Pointer to root tablet Stored in Chubby Open() 3/2/2021 Google, Inc. 34

Tablet Representation • Commit log (stored on GFS) – Redo log – Buffered in memory (at tablet server) • A set of locality groups • Each locality group = a set of SSTable files (stored on GFS) – SSTable = immutable 2 -level B-tree – Structured keys <row, column, timestamp> 3/2/2021 Google, Inc. 35

Tablet Representation read write buffer in memory (random-access) tablet server commit log SSTable file GFS 3/2/2021 Google, Inc. 36

Implementation Details • Garbage collection of files – Mark-sweep • Policies for when to write SSTables – Tradeoffs between recovery time, memory consumption, disk traffic • • Recovery Tablet-server membership Master election Support for compression 3/2/2021 Google, Inc. 37

Example Applications • Batch processing – Google. com - web indexing – Google Earth - image processing – Google Analytics - stats aggregation • User-Facing Applications – Orkut. com - social network – Personalized Search - query & click history – Writely - collaborative editing 3/2/2021 Google, Inc. 38

Sample Clusters Project Size (TB) Cells (billions) In Memory Crawl 800 1000 0% Google Analytics 200 80 0% Google Earth 0. 5 8 33% Google Earth 70 9 0% Personalized Search 4 6 5% Data from August 2006 3/2/2021 Google, Inc. 39

Usage • >500 clusters, >50 K tablet servers • 34 clusters of >400 tablet servers – 38 K machines in those clusters – 3. 7 M requests/second – 3 GB/s writes, 15 GB/s reads Data as of October 2007 3/2/2021 Google, Inc. 40

Current/future Issues • Reduce maintenance overhead – People are expensive, and they make mistakes – Lots of knobs to twiddle • Make the system more transparent – How do users understand/debug performance? • Increase machine utilization – Improve isolation within shared bigtable cells • Higher-level abstractions – Secondary indices 3/2/2021 Google, Inc. 41

Design Lessons • Solve application programmer needs – Build general solutions, but don’t over-abstract – Don’t be afraid to say no • Keep everything as simple as possible – Users will eventually misuse every part of the API – Users and faulty hardware will eventually exercise every code path • Design for change – We have rewritten almost every major subsystem – Building blocks evolve • Scale is interesting – Wide-area distributed storage is just one problem 3/2/2021 Google, Inc. 42