Distributed Systems Message Ordering Papadakis Harris Department of

Distributed Systems Message Ordering Papadakis Harris Department of Informatics Engineering TEI of Crete

Contents • Reliable Multicast • Ordered Multicast • Total ordering • Causal ordering • FIFO ordering

Reliable Multicast • Integrity: A correct process p in a group(m) delivers a message m at most once. • Safety property: a message delivered is identical to what was sent • Validity: If a correct process sends message m, , then it will eventually deliver m. • Guarantees liveness to the sender. • Liveness property: each message is finally delivered to the destination. • Agreement: If a correct process delivers message m, then all the other correct processes in group(m) will eventually deliver m.

Reliable Multicast Algorithm R-multicast B-multicast reliable unicast “USES”

Reliable Multicast Algorithm (R-multicast) Integrity Agreement Integrity, Validity if some correct process B-multicasts a message m, then, all correct processes deliver m too. If no correct process B-multicasts m, then no correct processes deliver m.

Ordered Multicast • FIFO ordering: If a correct process issues multicast(g, m) and then multicast(g, m’), then every correct process that delivers m’ will have already delivered m. • Causal ordering: If multicast(g, m) multicast(g, m’) then any correct process that delivers m’ will have already delivered m. • Total ordering: If a correct process delivers message m before m’, then any other correct process that delivers m’ will have already delivered m.

Total, FIFO and Causal Ordering • Totally ordered messages T 1 and T 2. • FIFO-related messages F 1 and F 2. • Causally-related messages C 1 and C 3 • Causal ordering implies FIFO ordering • Total ordering does NOT imply causal ordering. • Causal ordering does NOT imply total ordering. • Hybrid mode: causal-total ordering, FIFO-total ordering Totally-ordered FIFO-ordered Causalordered

FIFO-ORDERED MULTICAST

FIFO- Ordered Multicast v Processes the sent messages from each process in the order they were sent: v Each process keeps a sequence number for each other process. v Messages are sent from each process with a local sequence number v When a message is received, then: next expected, accept If #message is higher than expected, buffer lower than expected, reject

FIFO- Ordered Multicast Message processing FO-deliver Hold-back queue Incoming messages Delivery queue When delivery guarantees are met

FIFO Ordering Implementation • Spg: the number of messages p has sent to g. • Rqg: the sequence number of the latest message that p has delivered to group g. • For p to FO-multicast m to g • p increments Spg by 1 • p adds the value Spg to the message. • p B-multicasts message m to g. • Process p, upon receipt of message m from q with sequence number S: • p checks whether S= Rqg+1. If so, p FO-delivers m and increments Rqg • If S > Rqg+1, p places the message in the hold-back queue until the intervening messages have been delivered AND S= Rqg+1. • If S < Rqg+1, then ignores the message.

Example: FIFO Multicast Time P 1 P 2 P 3 000 100 200 1 2 1 Reject: 1<1+1 Accept: 2=1+1 210 200 Reject: Accept 1 1<=10++11 1 000 Accept 1=0+1 100 200 000 R 2 g Sequence Vector for P 1 R 3 g 210 Accept: 1=0+1 Buffer 2>0+1 S 1 g 1 1 100 210 000 R 1 g S 2 g Accept Buffer 2=1+1 Sequence Vector for P 2 R 3 g

CAUSAL-ORDERED MULTICAST

Causal Multicast • Assume g is a multicast group • Each process iєg keeps a vector Vgi of length |g| where • Vgi[j] counts the number of messages in group g from process j to i • Messages that process i multicasts contain Vgi • Remember the rule for receiving vector clock timestamps Vreceiver[j] = Max(Vreceiver[j] , Vmessage[j]), if not j Vreceiver[j] + 1 else • when process i receives a <m, Vgj> from j, then • Vgi[k] = max(Vgi[k], Vgj[k]) • Vgi[k] = Vgi[k] + 1 if k ≠ i if k = i • Remember: V(a) < V(b) if and only if a -> b

Causal Ordering using vector timestamps The number of messages that process j has received from i until now Guarantees Causal ordering

Example: Causal Ordering Multicast Reject: Accept P 1 1, 0, 0, 0 (1, 0, 0) P 2 0, 0, 0 1, 1, 0 (1, 1, 0) 0, 0, 0 (1, 1, 0) 1, 1, 0 (1, 0, 0) (1, 1, 0) P 3 1, 1, 0 Accept 1, 0, 0 1, 1, 0 Accept Buffer missing P 1(1) (1, 1, 0) >(1, 0, 0) Time Accept Buffered message

TOTAL-ORDERED MULTICAST

1 st Method- Using a Sequencer • Similar with FIFO • Each process keeps a sequence number for each multicast (as opposed to for each process) • The sender includes the unique id 'i' in each message m and sends <m, i> to the process sequencer (g) and to the group g • The Sequencer maintains an increasing number for each Sg multicast (contiguous and crescent) and B-multicasts the message (I, Sg) in g

1 st Method- Using a Sequencer unique msg id <m, i> P 1 sequencer Sg , Sg+1 <m, i> <order, i, Sg> P 2 P 3 rg rg Group g: P 1, P 2, P 3 Sequencer (g) : - Single point of failure - Bottleneck

2 nd Method: Totally Ordered multicast by buffering messages • Assumptions: Reliable, FIFO channel • A multicast message is also sent to the sender • Process pi multicasts m by timestamping it with the local clock value • Process j (including i) receives m • Places it in queue ordered in ascending timestamp order (use sender process id to break ties) • Multicasts a time-stamped acknowledgment (Note that timestamp(ack) > timestamp(m)) • Process j (including i) delivers m if • The m is the first in the queue • There is at least one message mi from every other process i in the queue, such as timestamp(m) < timestamp(mi)