Distributed Systems CS 15 440 Consistency and Replication

Distributed Systems CS 15 -440 Consistency and Replication – Part III Lecture 14, Oct 24, 2016 Mohammad Hammoud

Today… § Last Session § Consistency and Replication – Part II § Client-Centric Consistency Models § Today’s Session § Consistency and Replication – Part III § Replica Management & Consistency Protocols § Programming Models (Intro) § Announcements: § P 3 will be posted by tomorrow. It is due on Nov 14 § Your virtual clusters are ready- we will show you how to use them during the recitation 2

Overview Consistency Models Data-centric Consistency Models Client-centric Consistency Models Replica Management Replica Server Placement Content Replication and Placement Consistency Protocols Primary-based protocols Replicated-write protocols Cache-coherence protocols 3

Summary of Consistency Models Data-centric Models for Specifying Consistency Continuous Consistency Model Client-centric Models for Consistent Ordering of Operations Sequential Consistency Model Causal Consistency Model Eventual Consistency Monotonic Reads Client Consistency Guarantees Monotonic Reads Read your writes 4 Write follow reads

Overview Consistency Models Data-centric Consistency Models Client-centric Consistency Models Replica Management Replica Server Placement Content Replication and Placement Consistency Protocols Primary-based protocols Replicated-write protocols Cache-coherence protocols 5

Replica Management Replica management describes where, when and by whom replicas should be placed We will study two problems under replica management 1. Replica-Server Placement Decides the best locations to place the replica servers that can host data-stores 2. Content Replication and Placement Finds the best server for placing the contents 6

Overview Consistency Models Data-centric Consistency Models Client-centric Consistency Models Replica Management Replica Server Placement Content Replication and Placement Consistency Protocols Primary-based protocols Replicated-write protocols Cache-coherence protocols 7

Replica Server Placement Factors that affect placement of replica servers: What are the possible locations where servers can be placed? Should we place replica servers close-by or distribute them uniformly? How many replica servers can be placed? What are the trade-offs between placing many replica servers vs. few? How many clients are accessing the data from a location? More replicas at locations where most clients access improves performance and fault-tolerance If K replicas have to be placed out of N possible locations, find the best K out of N locations(K<N) 8

Replica Server Placement – An Example Approach Problem: K replica servers should be placed on some of the N possible replica sites such that Clients have low-latency/high-bandwidth connections A possible Greedy Approach: C=100 1. Evaluate the cost of placing a replica on each of the N potential sites Examining the cost of C clients connecting to the replica Cost of a link can be 1/bandwidth or latency R 1 C=40 RR 22 2. Choose the lowest-cost site C=90 C=60 R 3 3. Do the same every time you need to add a replica site for a new group or old sub-group of clients 9 (could lead to sub-optimal locations) R 4

Overview Consistency Models Data-centric Consistency Models Client-centric Consistency Models Replica Management Replica Server Placement Content Replication and Placement Consistency Protocols Primary-based protocols Replicated-write protocols Cache-coherence protocols 10

Content Replication and Placement In addition to the server placement, it is important to know: How, when and by whom different data items (contents) are placed on possible replica servers Identify how webpage replicas are replicated: Primary Servers in an organization Replica Servers on external hosting sites Permanent Replicas Server-initiated Replicas Client-initiated Replicas 11

Logical Organization of Replicas Permanent Replicas Server-Initiated Replicas Client-initiated Replicas Clients Server-initiated Replication Client-initiated Replication 12

1. Permanent Replicas Permanent replicas are the initial set of replicas that constitute a distributed data-store They are typically small in number There can be two types of permanent replicas: Primary replicas One or more servers in an organization Whenever a request arrives, it is forwarded into one of the primary replicas Mirror sites Geographically spread, and replicas are generally statically configured Clients pick one of the mirror sites to download the data 13

2. Server-initiated Replicas A third party (provider) owns the secondary replica servers, and they provide hosting service The provider has a collection of servers across the Internet The hosting service dynamically replicates files on different servers E. g. , Based on the popularity of a file in a region The permanent server chooses to host the data item on different secondary replica servers The scheme is efficient when updates are rare Examples of Server-initiated Replicas in Content Delivery Networks (CDNs) 14

Dynamic Replication in Server-initiated Replicas Dynamic replication at secondary servers: Helps to reduce the server load and improve client performance But, replicas have to dynamically push the updates to other replicas Rabinovich et al. [3] proposed a distributed scheme for replication: Each server keeps track of: i. ii. The closest server to any requesting client The number of requests per file per closest server E. g. , each server Q keeps track of cnt. Q(P, F) which denotes how many requests arrived at Q which are closer to server P (for a file F) If cnt. Q(P, F) > 0. 5 * cnt. Q(Q, F) Request P to replicate a copy of file F If cnt. P(P, F) < LOWER_BOUND Delete the file at replica Q If some other server is nearer to the clients, request replication over that server If the replication is not popular, delete the replica 15

3. Client-initiated Replicas Client-initiated replicas are known as client caches Client caches are used only to reduce the access latency of data e. g. , Browser caching a web-page locally Typically, managing a cache is entirely the responsibility of a client Occasionally, data-store may inform client when the replica has become stale 16

Summary of Replica Management Replica management deals with placement of servers and content for improving performance and fault-tolerance Replica Management Permanent Replicas Server Initiated Replicas Client Initiated Replicas So far, we know: • The required consistency models for applications • How to place replica servers and content What else do we need to provide consistency in a distributed system? 17

Overview Consistency Models Data-centric Consistency Models Client-centric Consistency Models Replica Management Replica Server Placement Content Replication and Placement Consistency Protocols Primary-based protocols Replicated-write protocols Cache-coherence protocols 18

Consistency Protocols A consistency protocol describes the implementation of a specific consistency model We are going to study three consistency protocols: Primary-based protocols One primary coordinator is elected to control replication across multiple replicas Replicated-write protocols Multiple replicas coordinate together to provide consistency guarantees Cache-coherence protocols A special case of client-controlled replication 19

Overview of Consistency Protocols Primary-based Protocols Replicated-Write Protocols Cache Coherence Protocols 20

Primary-based protocols In Primary-based protocols, a simple centralized design is used to implement consistency models Each data-item x has an associated “Primary Replica” The primary replica is responsible for coordinating write operations We will study one example of Primary-based protocols that implements Sequential Consistency Model Remote-Write Protocol 21

Remote-Write Protocol Rules: All write operations are forwarded to the primary replica Read operations are carried out locally at each replica Approach for write ops: (Budhiraja et al. 1993) Client connects to some replica RC If the client issues write operation to RC: RC forwards the request to the primary replica RP RP updates its local value RP forwards the update to other replicas Ri Other replicas Ri update, and send an ACK back to RP After RP receives all ACKs, it informs RC that the write operation is completed RC acknowledges the client, which in return completes the write operation x+=5 Client 1 Primary server R 1 R 2 R 3 =5 x 1=0 x 22=0 =5 =5 x 3=0 Data-store 22

Remote-Write Protocol – Discussion The Remote-Write protocol provides A simple way to implement sequential consistency Guarantees that clients see the most recent write operations However, latency is high in Remote-Write Protocols Clients block until all the replicas are updated Can a non-blocking strategy be applied? Remote-Write Protocols are applied to distributed databases and file systems that require fault-tolerance Replicas are placed on the same LAN to reduce latency 23

Overview of Consistency Protocols Primary-based Protocols Replicated. Write Protocols Cache Coherence Protocols Remote-Write Protocol 24

Replicated-Write Protocol In a replicated-write protocol, updates can be carried out at multiple replicas We will study one example on replicated-write protocols called Active Replication Protocol Here, clients write at any replica The modified replica will propagate updates to other replicas 25

Active Replication Protocol When a client writes at a replica, the replica will send the write operation updates to all other replicas Challenges with Active Replication Ordering of operations cannot be guaranteed across the replicas x+=2 x*=3 Client 1 Client 2 R 1 W(x) x+=2 R(x)0 R 2 R 1 R 2 R 3 xx 11=0 =2 =6 =0 =2 xx 22=6 x 3=0 =2 =6 Data-store R 3 R(x)6 W(x) R(x)2 26 x*=3 R(x)2 R(x)6

Centralized Active Replication Protocol Approach There is a centralized coordinator called the sequencer (Seq) When a client connects to a replica RC and issues a write operation RC forwards the update to the Seq assigns a sequence number to the update operation RC propagates the sequence number and the operation to other replicas Operations are carried out at all the replicas in the order defined by the sequencer x+=5 x-=2 Client 1 Client 2 10 R 1 Seq 10 R 2 x+=5 11 R 3 11 x-=2 27 Data-store

Overview of Consistency Protocols Primary-based Protocols Replicated-Write Protocols Remote-Write Protocols Active Replication Protocol Cache Coherence Protocols 28

Cache Coherence Protocols Caches are special types of replicas Typically, caches are client-controlled replicas Cache coherence refers to the consistency of data stored in caches How are the cache coherence protocols in sharedmemory multiprocessor (SMP) systems different from those in Distributed Systems? Coherence protocols in SMP assume cache states can be broadcasted efficiently In DS, this is difficult because caches may reside on different machines 29

Cache Coherence Protocols (Cont’d) Cache Coherence protocols determine how caches are kept consistent Caches may become inconsistent when a data item is modified: 1. 2. at the server replicas, or at the cache 30

When Data is Modified at the Server Two approaches for enforcing coherence: 1. Server-initiated invalidation Here, server sends all caches an invalidation message (when data item is modified) 2. Server updates the cache Server will propagate the update to the cache 31

When Data is Modified at the Cache The enforcement protocol may use one of three techniques: i. Read-only cache The cache does not modify the data in the cache The update is propagated to the server replica ii. Write-through cache Directly modify the cache, and forward the update to the server iii. Write-back cache The client allows multiple writes to take place at the cache The client batches a set of writes, and will send the batched write updates to the server replica 32

Summary of Consistency Protocols Primary-based Protocols Replicated-Write Protocols Cache Coherence Protocols Remote-Write Protocols Active Replication Protocol Coherence Enforcement Strategies 33

Consistency and Replication – A Very Brief Summary Replication improves performance and fault-tolerance However, replicas have to be kept reasonably consistent Consistency Models • A contract between the data-store and processes • Types: Data-centric and Client-centric Replication Management • Describes where, when and by whom replicas should be placed • Types: Replica Server Placement, Content Replication and Placement Consistency Protocols • Implement Consistency Models 34 • Types: Primary-based, Replicated-Write, Cache Coherence

New Topic + Programming Models- Part I 35

Objectives Discussion on Programming Models Why parallelism? Parallel computer architectures Traditional models of parallel programming Types of Parallel Programs Message Passing Interface (MPI) Map. Reduce, Pregel and Graph. Lab

Amdahl’s Law § We parallelize our programs in order to run them faster § How much faster will a parallel program run? § Suppose that the sequential execution of a program takes T 1 time units and the parallel execution on p processors takes Tp time units § Suppose that out of the entire execution of the program, s fraction of it is not parallelizable while 1 -s fraction is parallelizable § Then the speedup (Amdahl’s formula): 37

Amdahl’s Law: An Example § Suppose that 80% of your program can be parallelized and that you use 4 processors to run your parallel version of the program § The speedup you can get according to Amdahl’s law is: § Although you use 4 processors you cannot get a speedup more than 2. 5 times! 38

Real Vs. Actual Cases § Amdahl’s argument is too simplified to be applied to real cases § When we run a parallel program, there a communication overhead and a workload imbalance among processes (in general) Serial Parallel 20 20 80 Serial Parallel 20 Process 1 Process 2 20 20 80 20 Cannot be parallelized Process 3 Process 4 Cannot be parallelized Can be parallelized Process 3 Can be parallelized Process 4 Communication overhead Load Unbalance 1. Parallel Speed-up: An Ideal Case 2. Parallel Speed-up: An Actual Case

Guidelines § In order to efficiently benefit from parallelization, we can follow some guidelines like: 1. Maximize the fraction of your program that can be parallelized 2. Balance the workload of parallel processes 3. Minimize the time spent for communication 40

Next Class Continue with Programming Models 41