Dynamo Amazons Highly Available Keyvalue Store Adopted from

Dynamo: Amazon’s Highly Available Key-value Store Adopted from slides and/or materials by paper authors (Giuseppe De. Candia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall and Werner Vogels) 1

Motivation n Even the slightest outage has significant financial consequences and impacts customer trust. n The platform is implemented on top of an infrastructure of tens of thousands of servers and network components located in many datacenters around the world. n Persistent state is managed in the face of these failures - drives the reliability and scalability of the software systems 2

Motivation (Cont’d) n Build a distributed storage system: n Scale n Simple data model (name, value) n key-value store (big hashed table) n Highly available (sacrifice consistency) n Guarantee Service Level Agreements (SLA) 3

Data Model n Pros: n DB = Tables <-- Items <-- Attributes n Don’t require schema to work with the data n Have concept of primary key n Attribute = (name, value) n Value can be single or multiple (set). n Cons: n Don’t support cross-table joins n developers must be aware of foreign key concepts in the design themselves § No concept of embedding or referencing like Mongo 4

Data Model 5

Data Model n Primary Key • Hash Type Primary Key one attribute (called hash attribute) • Dynamo. DB builds • • an unordered hash index on this primary key attribute. • Why hashed index? • Index can be distributed to multiple servers 6

Data Model n Primary Key • Hash and Range Type Primary Key two attributes: hash attribute and range attribute. • Dynamo. DB builds 2 indexes as follows: • • an unordered hash index on the hash primary key attribute (for equality comparison) and • a sorted range index on the range primary key attribute (also support inequality comparison, e. g. , range queries). 7

Data Model n Primary Key Table Name Primary Key Type Hash Attribute Name Range Attribute Name Forum ( Name, . . . ) Hash Attribute Name: Name - Thread Hash and Range (Forum. Name, Subje ct, . . . ) Attribute Name: Forum. Name Attribute Name: Subject Reply Hash and Range ( Id, Reply. Date. Time, . . . ) Attribute Name: Id Attribute Name: Reply. Date. Time 8

Data Model n Primary Key • Design consideration • don’t support cross-table joins. • Example: • the Reply table stores both the forum name and subject values in the Id attribute. If you have a thread reply item, you can then parse the Id attribute to find the forum name and subject and use the information to query the Thread or the Forum tables. 9

System Assumptions and Requirements Query Model n simple read and write operations to a data item that is uniquely identified by a key. n Most of Amazon’s services can work with this simple query model and do not need any relational schema. n targeted applications - store objects that are relatively small (usually less than 1 MB) 10

ACID Properties (Cont’d) n Experience at Amazon has shown that data stores that provide ACID guarantees tend to have poor availability. n Dynamo targets applications that operate with weaker consistency (the “C” in ACID) if this results in high availability. 11

Efficiency n latency requirements which are in general measured at the 99. 9 th percentile of the distribution n Average performance is not enough 12

Service Level Agreements (SLA) n Application can deliver its functionality in bounded time n Every dependency in the platform needs to deliver its functionality with even tighter bounds. n Example n service guaranteeing that it will provide a response within 300 ms for 99. 9% of its requests for a peak client load of 500 requests per second. 13

Service-oriented architecture 14

Design Consideration n Sacrifice strong consistency for availability n Conflict resolution is executed during read instead of write, i. e. “always writeable”. 15

Design Consideration (Cont’d) n Incremental scalability. n Symmetry n Every node in Dynamo should have the same set of responsibilities as its peers. n Decentralization. n In the past, centralized control has resulted in outages and the goal is to avoid it as much as possible. n (Capacity) Heterogeneity. n This is essential in adding new nodes with higher capacity without having to upgrade all hosts at once. 16

Design Consideration (Cont’d) n “always writeable” data store where no updates are rejected due to failures or concurrent writes. n an infrastructure within a single administrative domain where all nodes are assumed to be trusted. 17

System architecture n Ring Partitioning n High Availability for writes n Handling temporary failures n Recovering from permanent failures n Membership and failure detection 18

Partition Algorithm n Consistent hashing: The output range of a hash function is treated as a fixed circular space or “ring”. n Replication: An item can be stored in multiple consecutive nodes depending on the degree of replication (N). In this example, N = 3. n ”Virtual Nodes”: Each node can be responsible for more than one virtual node. 19

Partition Algorithm n Example: n n a Node X has 3 virtual nodes If X becomes unavailable, n n the load handled by node X is evenly dispersed across some of the remaining nodes. Key a node Z Key b node Y Key c node W W X Z X b c X Z Y 20

Partition (Cont’d) n Advantages of using virtual nodes: n If a node becomes unavailable，the load handled by this node is evenly dispersed across the remaining available nodes. n When a node becomes available again, or a new node is added to the system, the newly available node accepts a roughly equivalent amount of load from some of the other available nodes. n The number of virtual nodes that a node is responsible can be decided based on its capacity, accounting for heterogeneity in the physical infrastructure. 21

Replication n Each data item is replicated at N hosts. n “preference list”: The list of (distinct physical) nodes that is responsible for storing a particular key. 22

Data Versioning n A put() call may return to its caller before the update has been applied at all the replicas n A get() call may return many versions of the same object. n Key Challenge: distinct version sub-histories - need to be reconciled. n Solution: uses vector clocks in order to capture causality between different versions of the same object. 23

Vector Clock n A vector clock is a list of (node, counter) pairs. n Every version of every object is associated with one vector clock. n If the counters on the first object’s clock are less-than-or-equal to all of the nodes in the second clock, then the first is an ancestor of the second and can be forgotten. 24

Vector clock example 25

Vector clock n In case of network partitions or multiple server failures, write requests may be handled by nodes that are not in the top N nodes in the preference list causing the size of vector clock to grow. n Dynamo stores a timestamp that indicates the last time the node updated the data item. n When the number of (node, counter) pairs in the vector clock reaches a threshold (say 10), the oldest pair is removed from the clock. 26

Execution of get () and put () operations n Two strategies to select a node: 1. Route its request through a generic load balancer that will select a node based on load information. 2. Use a partition-aware client library that routes requests directly to the appropriate coordinator (first) nodes. 27

Execution of get () and put () operations (Cont’d) n The advantage of the first approach is that the client does not have to link any code specific to Dynamo in its application n The second strategy can achieve lower latency because it skips a potential forwarding step. 28

Temporary Failures Sloppy Quorum n R/W is the minimum number of nodes that must participate in a successful read/write operation. n Setting R + W > N yields a quorum-like system. n In this model, the latency of a get (or put) operation is dictated by the slowest of the R (or W) replicas. For this reason, R and W are usually configured to be less than N, to provide better latency. 29

Hinted handoff n Assume N = 3. When A is temporarily down or unreachable during a write, send replica to D. n D is hinted that the replica is belong to A and it will deliver to A when A is recovered. n Again: “always writeable” 30

Summary of techniques used in Dynamo and their advantages Problem Technique Advantage Partitioning Consistent Hashing Incremental Scalability High Availability for writes Vector clocks with reconciliation during reads Version size is decoupled from update rates. Handling temporary failures Sloppy Quorum and hinted handoff Provides high availability and durability guarantee when some of the replicas are not available. Recovering from permanent failures Anti-entropy using Merkle trees Membership and failure detection Gossip-based membership protocol and failure detection. Synchronizes divergent replicas in the background. Preserves symmetry and avoids having a centralized registry for storing membership and node liveness information. 31

Implementation n Java n Local persistence component allows for different storage engines to be plugged in: Berkeley Database (BDB) Transactional Data Store: object of tens of kilobytes n My. SQL: object of > tens of kilobytes n BDB Java Edition, etc. n 32

Conclusion n Dynamo is a highly available and scalable data store for Amazon. com’s e-commerce platform. n Dynamo has been successful in handling server failures, data center failures and network partitions. n Dynamo is incrementally scalable and allows service owners to scale up and down based on their current request load. 33

Conclusion (Cont’d) n Dynamo is incrementally scalable and allows service owners to scale up and down based on their current request load. n Dynamo allows service owners to customize their storage system by allowing them to tune the parameters N, R, and W. 34