Synchronization Chapter 5 Synchronization 1 Synchronization q Multiple

Synchronization Chapter 5 Synchronization 1

Synchronization q Multiple processes must not simultaneously access shared resource q Ordering may be important o Such as, msg 1 must come before msg 2 q Time o Absolute time vs relative time q May want one process to coordinate o Election algorithms Chapter 5 Synchronization 2

Synchronization q Special topics… q Distributed mutual exclusion o Protect shared resources from simultaneous access q Distributed transactions o Similar, but try to optimize access thru “advanced concurrency control” Chapter 5 Synchronization 3

What Time is It? q Easy to answer in a non-dist system o Spse A asks for time, then B o B’s time will be later than A’s o In dist system, this may not be true Spse A checks time, then B q B’s time might not be later than A’s q o That is, time on A and B might not agree o If time comes from a central location, network communication variation is a problem Chapter 5 Synchronization 4

What Time is It? Why do we care about time? q Consider make example q Make used to compile and link multiple source files into one executable file q If file. o was last modified before file. c, then file. c must be recompiled q If file. o was last modified after file. c, then no need to recompile file. c q This breaks if time is not the same in distributed system q Chapter 5 Synchronization 5

Clock Synchronization q Both machines have their own clock o Clocks differ by “ 2” What will make do with output. c? q Oops! q Chapter 5 Synchronization 6

Time q With single processor system o Doesn’t matter if time is incorrect o Relative time is what’s important q If more than one processor o Clock skew is inevitable q Multiple clock problems o How to synchronize with “real” clock? o How to synchronize clocks with each other? q But first we digress… Chapter 5 Synchronization 7

Physical Clocks q Time between 2 transits of the sun o Solar day q Solar second is 1/86400 th solar day Chapter 5 Synchronization 8

Physical Clocks q Period of earth rotation not constant o Earth is slowing due to drag o Days are getting longer Atomic clock invented 1948 q Official second is now q o 9, 192, 631, 770 transitions of cesium 133 International Atomic Time (TAI) q Today, 86, 400 TAI seconds is about 3 msec less than mean solar day! q Chapter 5 Synchronization 9

Physical Clocks Solar seconds are not of constant length q TAI seconds are of constant length q o Leap seconds are used to keep in phase with sun o Add leap second when discrepancy > 800 msec q Otherwise noon would eventually be before breakfast might cause riots! Chapter 5 Synchronization 10

Physical Clocks q TAI with leap seconds is known as o Universal Coordinated Time (UTC) UTC replaces Greenwich Mean Time (GMT) q NIST operates radio WWV from Colorado q o Sends out pulse at start of each UTC second o But only accurate to within 1 msec o Do to atmospheric effects, can vary by 10 msec Some satellites offer similar service q In any case, must know relative position q o To compensate for propagation delay Chapter 5 Synchronization 11

Clock Sync. Algorithms q Suppose one machine monitor WWV q How to keep other clocks in sync? o Let t be UTC time o Let Cp(t) be time on machine p q Ideally, want Cp(t) = t o We’ll be happy if d. Cp/dt = 1 Chapter 5 Synchronization 12

Clock Sync. Algorithms q q Clocks drift Suppose One clock is slow and one is fast… Drift apart at twice the drift rate Chapter 5 Synchronization 13

Clock Sync. Algorithms Let Cp(t) be time on machine p q Ideally, want Cp(t) = t q o Or d. Cp/dt = 1 q But processor clocks can drift o If maximum rate of drift is o After t, two clocks could be 2 t apart q If you want clocks to differ by less than o Must synchronize clocks every / 2 seconds q How to synchronize? Chapter 5 Synchronization 14

Clock Sync. Algorithms How to synchronize clocks? q Cristian’s algorithm q o Pull protocol q Berkeley algorithm o Push protocol q Averaging algorithms o Decentralized approach Network Time Protocol (NTP) q Multiple external time sources q Chapter 5 Synchronization 15

Cristian's Algorithm q Spse time server has WWV time q Clients want to stay within of others q Every / 2 seconds or less… o Client asks time server for time q Somebody got an algorithm named after themselves for that? q See next slide Chapter 5 Synchronization 16

Cristian's Algorithm q What are the potential problems? o Time cannot run backwards o Takes (variable) time to get reply Chapter 5 Synchronization 17

Cristian's Algorithm q Time cannot run backwards o If clock is fast… o Increment time more slowly than usual q Must account for time to get reply o How to do this? o Educated guess! Roundtrip time divided by 2 o Account for time server takes to process, multiple roundtrip measurements, etc. Chapter 5 Synchronization 18

Berkeley Algorithm q Cristian’s “algorithm” q Berkeley algorithm o Time server is passive o o o Time server is aggressive Does not require server to know UTC Server polls clients Computes average time Pushes result to clients Chapter 5 Synchronization 19

Berkeley Algorithm a) b) c) Server asks others for their clock values Machines answer Server tells others how to adjust their clock Chapter 5 Synchronization 20

Averaging Algorithms q Cristian’s and Berkeley are centralized q Averaging (decentralized) approach… o All machines broadcast time o Everybody computes average o The usual refinements apply q When to broadcast? q Only practical on a LAN Chapter 5 Synchronization 21

Network Time Protocol q According to book, NTP uses o “advanced clock synchronization algorithms” o Accuracy range of 1 to 50 msec But NTP is not very secure q NTP actually uses Marzullo’s Algorithm q o Aka the Intersection Algorithm q Have a collection of times intervals o Example: time of 10 2 gives interval [8, 12] Chapter 5 Synchronization 22

Network Time Protocol q Given collection of times intervals o Of the form [a, b] q Marzullo’s algorithm finds consistent interval o Efficient: linear in time and space o If no consistent interval, finds interval(s) consistent with the most sources Marzullo takes center of resulting interval q Intersection Algorithm refines this q o Use statistical info on confidence intervals o Selected time not necessarily midpoint Chapter 5 Synchronization 23

Multiple External Time Sources q Suppose very accurate time needed q Multiple UTC sources? q But these will not agree q So need to average (or similar) o Network delays o Processing delays, etc. q Not clear that this helps very much! Chapter 5 Synchronization 24

Use of Synchronized Clocks Today, computers can be at or near UTC q How to make use of this? q To enforce “at most once delivery” q Traditional approach q o Server keeps track of msg numbers o Checks list against incoming msg numbers o How long to keep list? What if server crashes? q Alternative is to use timestamps o We discuss other apps in later sections Chapter 5 Synchronization 25

Logical Clocks q Usually good enough to agree on time o Even if it’s not the actual time q Often sufficient to agree on order o Recall make example q Lamport time o Synchronize logical clocks q Vector timestamps o Extension of Lamport’s algorithm Chapter 5 Synchronization 26

Lamport Timestamps q “Happens before”: a b q According to Tanenbaum: a b if all processes agree that a came before b q Lamport actually defines “ ” as the “smallest” relation satisfying o If a occurs before b on same processor then a b o If a == send, b == receive, a b o Transitive: a b and b c implies a c Chapter 5 Synchronization 27

Lamport Timestamps “Happens before”: a b q Does “happens before” equal “really happened before”? q If a and b are on same process and a occurs before b, then a b q If a == msg sent, b == (same) msg received, then a b q o It takes time for message to be sent q If a b and b a, msgs are concurrent / Chapter 5 Synchronization / 28

Lamport Timestamps q For event a, want timestamp C(a) o If a b then C(a) < C(b) o C is a non-decreasing function o Time cannot go backwards! q Lamport’s solution o Each msg carries timestamp with it o If local time is less than timestamp, set local time to timestamp + 1 o Advance clock between any two events q Illustrated on next slide… Chapter 5 Synchronization 29

Lamport Timestamps 0 0 0 8 10 6 8 10 12 16 20 18 24 30 18 24 24 32 40 30 40 50 36 48 60 36 48 42 56 70 42 61 70 64 80 48 69 80 54 72 90 70 77 90 60 80 100 76 85 100 6 48 a) A D B C Three processes with different clocks Chapter 5 Synchronization b) A D B C 30 60 Lamport's algorithm corrects the clocks 30

Lamport Timestamps q Can also insure that no two events ever occur at exactly the same time o 40. 1 for process 1 o 40. 2 for process 2, etc. q With this refinement, we have a total ordering on all events in the system o If a b on same process then C(a) < C(b) o If a == msg sent, b == msg received, then we have C(a) < C(b) o If a b then C(a) C(b) Chapter 5 Synchronization 31

Totally-Ordered Multicast q Consider replicated database o Spse replica in San Jose and in New York o Query goes to nearest copy q Updates are tricky o Must have updates in same order at replicas o For example: Interest calculation and deposit q For consistency, no “right” order o Just want updates to happen in same order q Correctness is a different story… Chapter 5 Synchronization 32

Non-Totally-Ordered Multicast Deposit q Interest Assumptions o $1000 in acct, deposit is $1000, interest rate is 10% q q On left, $2200, on right $2100 Inconsistent! Chapter 5 Synchronization 33

Totally-Ordered Multicast Assume msgs received in order and no loss q Using Lamport timestamps… q o o q Msgs timestamped with sender’s logical time Multicast sent to all sites, including sender Msgs go into local queue in timestamp order Multicast ACK msgs (to yourself too) Message only removed from queue if o It is at head of queue and o It has been ACKed q Does this work? See next slide… Chapter 5 Synchronization 34

Totally-Ordered Multicast q q $1000 in acct, deposit is $1000, interest rate 10% What happens in this case? Deposit Interest After 45… Deposit: 45 Later… Interest: 10 Deposit: 45 ACK(D): 46 ACK(I): 90 After 10… Interest: 10 30 45 Deposit Interest 0 10 60 20 75 45 90 ACK(I) ACK(D) 46 105 90 120 91 Chapter 5 Synchronization Later… Interest: 10 Deposit: 45 ACK(D): 46 ACK(I): 90 35

Totally-Ordered Multicast Deposit Interest After 45… Deposit: 45 Later… Interest: 10 Deposit: 45 ACK(D): 46 ACK(I): 90 q q After 10… Interest: 10 30 45 Deposit Interest 0 10 60 20 75 45 90 ACK(I) ACK(D) 46 105 90 120 91 Later… Interest: 10 Deposit: 45 ACK(D): 46 ACK(I): 90 When is interest calculation done at each replica? When is deposit made? Chapter 5 Synchronization 36

Scalar Timestamps q q Scalar timestamps (such as Lamport timestamps) give total ordering using C(a) But C(a) < C(b) does not mean that event a really happened before b 1 2 P 1 P 2 3 2 1 4 3 1 P 3 q 5 9 7 9 11 10 4 5 8 6 7 The “ 4” at P 2 occurs before the “ 3” at P 1 Chapter 5 Synchronization 37

Vector Timestamps q Lamport timestamps don’t reflect causality o Local events are causally ordered q Example: multicast news posting o Response might arrive before original posting Vector timestamps do reflect causality q Must specify q o Local data structures to represent logical time o Update mechanism/protocol q Tanenbaum’s description is confusing! Chapter 5 Synchronization 38

Vector Timestamps q Want vector timestamp such that o If VT(a) < VT(b) then a causally precedes b q Process Pi maintains vector Vi o Vi[i] is incremented for each event at i o Vi[j] is Pi’s current view of the number of events that have occurred at process Pj Vi[i] is easy to maintain q Vi[j] is obtained from info sent with msgs q o Each message includes vector timestamp Chapter 5 Synchronization 39

Vector Timestamps Suppose Pj received msg m from Pi q Pi includes it’s vector timestamp, vt q Then Pj adjusts its values according to vt q Pj then knows the number of events on which m can depend q Tanenbaum claims… q o Pj knows no. of messages it must receive before it has seen everything that m could depend on o Not true! Event msg! Chapter 5 Synchronization 40

Vector Timestamps 1 0 0 2 0 0 3 0 0 4 3 4 5 3 4 P 1 2 0 0 0 1 0 P 2 0 0 1 2 3 0 2 2 0 P 3 Chapter 5 Synchronization 5 3 4 2 4 0 2 3 4 2 3 3 2 5 6 4 5 5 4 2 3 4 41

Vector Timestamp Modified (useful) form of VT q Suppose Vi[i] counts msgs sent by Pi q Now consider multicast newsgroup q Suppose Pi post a message q o Includes vector vt(a) q Suppose Pj posts a response o Includes vector vt(b) q Want vt(a) to reflect msgs known to Pi when a was sent, and similarly for vt(b) Chapter 5 Synchronization 42

Vector Timestamp q Let Vi[i] be number of msgs sent by Pi o Vi[j] = number of messages Pi received from Pj Pi sends a with vt(a), later Pj sends b, vt(b) q Suppose Pk receives b before a q Pk waits to deliver msg b until q o vt(b)[j] == Vk[j] + 1 § This is the next msg expected from Pj o vt(b)[i] <= Vk[i], all i j § Ensures that Pk must have seen msg a Chapter 5 Synchronization 43

Vector Timestamp P 1 0 0 0 P 2 0 0 0 1 0 0 2 0 0 1 1 0 P 3 2 1 0 2 0 0 1 1 0 2 1 0 0 1 Queue 1 msg a 0 2 Queue 0 msg b 0 msg c 0 0 0 1 0 0 c Chapter 5 Synchronization 1 1 0 a 2 1 0 b 44

Global State q Global state of distributed system q Useful to know global state to o All local states plus msgs in transit o Definition of “state” can vary o Know that computation is finished o Detect deadlock q How to record global state? o Distributed snapshot Chapter 5 Synchronization 45

Global State q Distributed snapshot o A consistent state “in which the system might have been” o For example, if Q received msg from P then must show that P sent the msg o P sent msg Q has not yet received is OK q Global state represented by a cut q Next slide… Chapter 5 Synchronization 46

Global State a) b) Consistent cut Inconsistent cut Chapter 5 Synchronization 47

Global State Assume distributed system uses point-topoint unidirectional communication q Suppose P starts snapshot q o P records its state o P sends “marker” to neighbors q When Q receives marker o First marker on any channel: Q records state o Append incoming messages from S until marker from S is received o Q is done when it has received marker on all incoming channels Chapter 5 Synchronization 48

Global State q This figure does not match algorithm! q See next few slides… Chapter 5 Synchronization 49

Global State q Consider the following example q Bank has 3 branches, A, B, C q Each branch connected to others by o Unidirectional point-to-point links q State consists of o Money in branch and… o …money in transit between branches Chapter 5 Synchronization 50

Global State A Begin: SA Done: SA M 3 M 2 M 5 Done: SB B SB M 4 M 1 M 6 C Done: SC SC Note that no messages are in transit q Global state: (SA, SB, SC) q Chapter 5 Synchronization 51

Global State A Begin: SA (SA, T) M 2 T Done: (SA, T) M 3 M 5 Done: SB B SB M 4 M 1 M 6 C Done: SC SC Global state: (SA, T, SB, SC) q This does not work if msgs can be reordered! q Chapter 5 Synchronization 52

Global State q Example: Termination detection q Process Q received marker 1 st time o Process that sent it is Q’s predecessor o When Q completes its part… o …Q sends DONE msg to its predecessor q When is snapshot DONE? o When initiator of snapshot received DONE from all of its successors Chapter 5 Synchronization 53

Global State Problem: if DONE and msgs in transit, then computation may not really be done q Are msgs part of snapshot or computation? q Modification: send DONE provided q o All of Q’s successors returned DONE and o Q has not received any msg between time state was recorded and marker(s) received Otherwise send CONTINUE msg q DONE when initiator receives all DONEs q o If CONTINUEs, must do it again Chapter 5 Synchronization 54

Election Algorithms q May want one process to coordinate o We don’t care which process How to choose coordinator? q Have an election! q o o q Assume each process has a unique number All processes know everybody else’s number But some processes may be down Want to elect (live) process with highest number We’ll consider two election algorithms o Bully algorithm and ring algorithm Chapter 5 Synchronization 55

Bully Algorithm q P notices coordinator not responding o P sends ELECTION msg to all processes with higher number than P’s o If no one responds, P becomes coordinator o If a higher number responds, P is done q Process receives ELECTION from lower no. o Responds with OK o If not already doing so, it initiates an election q Eventually, everybody gives up… o Except for the biggest bully Chapter 5 Synchronization 56

Bully Algorithm q q Process 7 was coordinator until he died Process 4 is first to notice, so holds an election 5 and 6 respond, 4 gives up (why not stop here? ) Now 5 and 6 each hold an election Chapter 5 Synchronization 57

Bully Algorithm d) e) Process 6 tells 5 to give up Process 6 wins, then tells everyone Chapter 5 Synchronization 58

Ring Algorithm q Assume processes are ordered o Everyone knows their successor o Note that no “token” involved q Spse P notices coordinator has died o P sends ELECTION msg to its successor with P’s number attached o If no response, sends msg to P’s successor, and so on o Each guy in chain appends its number o When msg gets back to P, it selects highest number on list and sends COORDINATOR msg Chapter 5 Synchronization 59

Ring Algorithm 5 and 2 both initiate ELECTION q What will happen? q Chapter 5 Synchronization 60

Mutual Exclusion q Critical region a place where mutual exclusion is required o Example: update to a shared data structure q For single processor system o Use semaphore, monitors, etc. q Possible istributed system approaches o Imitate single processor approach o Distributed approach o Token ring approach Chapter 5 Synchronization 61

Centralized Algorithm Elect a coordinator q If P want to enter critical region q o Checks with coordinator q How does coordinator deny request? o Either explicit denial or no response o Queues any pending requests q Fair, efficient, etc. o No starvation? q But it’s centralized and we hate that! Chapter 5 Synchronization 62

Centralized Algorithm a) b) c) Process 1 OK to enter a critical region Process 2 asks permission to enter the same critical region, but no reply Process 1 exits, coordinator replies to 2 Chapter 5 Synchronization 63

Distributed Algorithm q For this, we need a total ordering on events o We know how to do this, right? q P wants to enter critical region o Send request msg (with timestamp) to everybody o Including itself q When request is received o Receiver not in critical region? Send OK o Receiver in critical region? No reply, queue request o Receiver wants to enter critical region but has not yet? Check timestamps, low one wins q After OKed by everybody, enter critical region Chapter 5 Synchronization 64

Distributed Algorithm a) b) c) Processes 0 and 2 want to enter critical region Process 0 has the lowest timestamp, it wins When process 0 is done, 2 gets its turn Chapter 5 Synchronization 65

Token Ring Algorithm q. A logical ring with a token q Token passed around ring q Process can only enter critical region when it has the token q Easy to see that this works! q Usual token ring problems apply Chapter 5 Synchronization 66

Token Ring Algorithm a) b) Unordered group of processes Logical ring (also need a token) Chapter 5 Synchronization 67

Comparison of Mutual Exclusion Algorithms Algorithm Messages per entry/exit Delay before entry (in message times) Problems Centralized 3 2 Coordinator crash Distributed 2(n– 1) Crash of any process Token ring 1 to 0 to n – 1 Lost token, process crash q ? ? ? Chapter 5 Synchronization 68

Distributed Transactions q Read the book! Chapter 5 Synchronization 69

Transaction Model q Updating master tape is fault tolerant Chapter 5 Synchronization 70

Transaction Model Primitive Description BEGIN_TRANSACTION Make the start of a transaction END_TRANSACTION Terminate the transaction and try to commit ABORT_TRANSACTION Kill the transaction and restore the old values READ Read data from a file, a table, or otherwise WRITE Write data to a file, a table, or otherwise q Primitives for transactions Chapter 5 Synchronization 71

The Transaction Model BEGIN_TRANSACTION reserve WP -> JFK; reserve JFK -> Nairobi; reserve Nairobi -> Malindi; END_TRANSACTION (a) a) b) BEGIN_TRANSACTION reserve WP -> JFK; reserve JFK -> Nairobi; reserve Nairobi -> Malindi full => ABORT_TRANSACTION (b) Transaction to reserve 3 flights commits Aborts when 3 rd flight unavailable Chapter 5 Synchronization 72

Distributed Transactions a) b) A nested transaction A distributed transaction Chapter 5 Synchronization 73

Private Workspace a) b) c) File index and disk blocks of 3 -block file After transaction modified 0, appended block 3 After committing Chapter 5 Synchronization 74

Writeahead Log x = 0; y = 0; BEGIN_TRANSACTION; x = x + 1; y=y+2 x = y * y; END_TRANSACTION; (a) Log Log [x = 0 / 1] [y = 0/2] [x = 1/4] (b) (c) (d) a) A transaction b) – d) Log before statement is executed Chapter 5 Synchronization 75

Concurrency Control q Managers for handling transactions Chapter 5 Synchronization 76

Concurrency Control q Chapter 5 Synchronization Managers for distributed transactions 77

Serializability BEGIN_TRANSACTION x = 0; x = x + 1; END_TRANSACTION BEGIN_TRANSACTION x = 0; x = x + 2; END_TRANSACTION (a) BEGIN_TRANSACTION x = 0; x = x + 3; END_TRANSACTION (b) (c) Schedule 1 x = 0; x = x + 1; x = 0; x = x + 2; x = 0; x = x + 3 Legal Schedule 2 x = 0; x = x + 1; x = x + 2; x = 0; x = x + 3; Legal Schedule 3 x = 0; x = x + 1; x = 0; x = x + 2; x = x + 3; Illegal (d) a) – c) Transactions T 1, T 2, and T 3 d) Possible schedules Chapter 5 Synchronization 78

Two-Phase Locking q Two-phase locking (duh!) Chapter 5 Synchronization 79

Two-Phase Locking q Strict two-phase locking Chapter 5 Synchronization 80

Pessimistic Timestamp Ordering q Concurrency control using timestamps Chapter 5 Synchronization 81