Building Dependable Distributed Systems Chapter 7 Wenbing Zhao

Building Dependable Distributed Systems Chapter 7 Wenbing Zhao Department of Electrical and Computer Engineering Cleveland State University wenbing@ieee. org Building Dependable Distributed Systems, Copyright Wenbing Zhao 1

Outline n Byzantine general problem q n Practical Byzantine fault tolerance q q n By Leslie Lamport, Robert Shostak, & Marshall Pease By Miguel Castro and Barbara Liskov, OSDI’ 99 http: //www. pmg. csail. mit. edu/papers/osdi 99. pdf Zyzzyva 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 2 Wenbing Zhao

The Byzantine Generals Problem n n n Abstract model of a computer system that may have faulty components Faulty components may send conflicting information to different parts of the system Scenario where Byzantine Generals must reach agreement in the presence of traitors 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 3 Wenbing Zhao

The Byzantine Generals Scenario General Commanding Lieutenant General Byzantine Army Division Enemy City General Lieutenant General Traitorous General Byzantine Army Division 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 4 Wenbing Zhao

Byzantine Generals Problem n A commanding general must send an order to his n-1 lieutenants such that q q n n n IC 1. All loyal lieutenants obey the same order IC 2. If the commanding general is loyal, then every loyal lieutenant obeys the order he sends IC 1 = Agreement clause IC 2 = Validity clause IC 1 and IC 2 are called interactive consistency Conditions 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 5 Wenbing Zhao

Byzantine Agreement Protocol n Assumption: q Every message that is sent is delivered correctly n q The receiver of a message knows who sent it n q Traitors cannot interfere with messages they do not sent Traitors cannot spoof messages The absence of a message can be detected n Traitors cannot prevent an agreement by not sending => Synchronous system + no spoofing 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 6 Wenbing Zhao

Byzantine Agreement Protocol (f=1) n n Round 1: the commander sends a value to each of the lieutenants Round 2: each of the lieutenants sends the value it received to its peers At the end of round 2, each lieutenant check to see if there is a majority opinion (attack or retreat). We have a solution if there is Question is: how many generals needed to tolerate f number of traitors? Note: f>=2 cases are much more complicated and expensive (Oral message algorithms) 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 7 Wenbing Zhao

Unsolvable Situations – N=3, f=1 Attack lieutenant Commander He said Retreat Attack lieutenant Commander Attack lieutenant 10/2/2020 Retreat He said Retreat EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao lieutenant 8 Wenbing Zhao

Byzantine Agreement Protocol (f=1) Commander Attack lieutenant He said Retreat lieutenant He said Attack He said Retreat He said Attack n Retreat lieutenant He said Attack If there are f traitors, then there must be at least 3 f + 1 total generals for IC 1 and IC 2 to hold 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 9 Wenbing Zhao

Byzantine Agreement Protocol (f=1) n n Under our assumption, if message digital signature is used and assuming the signature cannot be forged, we need only N=2 f+1 to tolerate f traitors q The commander still can send different information to different lieutenant, but a lieutenant cannot lie about what the commander has told him In asynchronous systems, N=2 f+1 is not sufficient q We have to stop after collecting f+1 input because the f faulty traitors could simply refrain from sending q Unfortunately there might be f inputs from traitors 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 10 Wenbing Zhao

Introduction to BFT Paper n n The growing reliance of industry and government on online information services Malicious attacks become more serious and successful More software errors due to increased size and complexity of software This paper presents “practical” algorithm for state machine replication that works in asynchronous systems like the Internet 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 11 Wenbing Zhao

Assumptions n n n Asynchronous distributed system The network may fail to deliver, delay, duplicate or deliver them out of order Faulty nodes may behave arbitrarily Independent node failures The adversary cannot delay correct nodes indefinitely All messages are cryptographically signed by their sender and these signatures cannot be subverted by the adversary 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 12 Wenbing Zhao

Service Properties n n n A (deterministic) service is replicated among ≥ 3 f+1 processors. Resilient to ≤ f failures Safety: All non-faulty replicas guaranteed to process the same requests in the same order Liveness: Clients eventually receive replies to their requests 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 13 Wenbing Zhao

Optimal Resiliency n n n Imagine non-faulty processors trying to agree upon a piece of data by telling each other what they believe the data to be A non-faulty processor must be sure about a piece of data before it can proceed f replicas may refuse to send messages, so each processor must be ready to proceed after having received (n-1)-f messages q Total of n-1 other replicas 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 14 Wenbing Zhao

Optimal Resiliency But what if f of the (n-1)-f messages come from faulty replicas? n To avoid confusion, the majority of messages must come from non-faulty nodes, i. e, (n-f-1)/2 ≥ f => Need a total of ≥ 3 f+1 replicas n 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 15 Wenbing Zhao

BFT Algorithm in a Nutshell Backup f + 1 Match (OK) Client Primary Backup 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 16 Wenbing Zhao

Replicas and Views Set of replicas (R): |R| ≥ 3 f + 1 R 0 R 1 R 2 ……… R|R-1| 0 View 1 For view v: primary p is assigned such that p= v mod |R| 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 17 Wenbing Zhao

Safeguards n n n If the client does not receive replies soon enough, it broadcasts the request to all replicas If the request has already been processed, the replicas simply re-send the reply (replicas remember the last reply message they sent to each client) If the primary does not multicast the request to the group, it will eventually be suspected to be faulty by enough replicas to cause a view change 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 18 Wenbing Zhao

Normal Case Operation Client {REQUEST, o, t, c} Primary o – Operation t – Timestamp c - Client Timestamps are totally ordered such that later requests have higher timestamps than earlier ones 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 19 Wenbing Zhao

Normal Case Operation n n Primary p receives a client request m , it starts a three-phase protocol Three phases are: pre-prepare, commit 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 20 Wenbing Zhao

Pre-Prepare Phase Backup Primary <<PRE-PREPARE, v, n, d> , m> v – view number n – sequence number d – digest of the message D(m) m – message 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao Backup 21 Wenbing Zhao

Prepare Phase n A backup accepts the PRE-PREPARE message only if: q q The signatures are valid and the digest matches m It is in view v It has not accepted a PRE-PREPARE for the same v and n Sequence number is within accepted bounds 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 22 Wenbing Zhao

Prepare Phase n If backup i accepts the pre-prepare message it enters prepare phase by multicasting <PREPARE, v, n, d, i> n n to all other replicas and adds both messages to its log Otherwise does nothing Replica (including primary) accepts prepare message and adds them to its log, provided that q Signatures are correct q View numbers match the current view q Sequence number is within accepted bounds 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 23 Wenbing Zhao

Prepare Phase n n At replica i, prepared (m, v, n, i) = true, iff 2 f PREPARE from different backups that match the pre-prepare When prepared = true, replica i multicasts <COMMIT, v, n, d , i> to other replicas 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 24 Wenbing Zhao

Agreement Achieved n n If primary is non-faulty then all 2 f+1 non-faulty replicas agree on the sequence number If primary is faulty q q n Either ≥f+1 non-faulty replicas (majority) agree on some other sequence and the rest realize that the primary is faulty Or, all non-faulty replicas will suspect the primary is faulty When a faulty primary is replaced, the minority of confused non-faulty replicas are brought up to date up by the majority 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 25 Wenbing Zhao

Commit Phase n n n Replicas accept commit messages and insert them in their log provided signatures are same Define committed and committed-local predicates as q Committed (m, v, n) = true, iff prepared (m, v, n, i) is true for all i in some set of f+1 non-faulty replicas q Committed-local (m, v, n, i) = true iff the replica has accepted 2 f+1 commit message from different replicas that match the pre-prepare for m If Committed-local (m, v, n, i) is true for some non-faulty replica i, then committed (m, v, n) is true 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 26 Wenbing Zhao

Commit Phase n n n Replica i executes the operation requested by m after committed-local (m, v, n, i) = true and i’s state reflects the sequential execution of all requests with lower sequence numbers The PRE-PREPARE and PREPARE phases of the protocol ensure agreement on the total order of requests within a view The PREPARE and COMMIT phases ensure total ordering across views 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 27 Wenbing Zhao

Normal Operation Reply n n All replicas sends the reply <REPLY, v, t, c, i, r>, directly to the client v = current view number t = timestamp of the corresponding request i = replica number r = result of executing the requested operation c = client id Client waits for f+1 replies with valid signatures from different replicas, and with same t and r, before accepting the result r 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 28 Wenbing Zhao

Normal Case Operation: Summery Request Pre-prepare Prepare Commit Reply C Primary: 0 1 2 Faulty: 3 10/2/2020 X EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 29 Wenbing Zhao

Garbage Collection n Used to discard messages from the log For the safety condition to hold, messages must be kept in a replica’s log until it knows that the requests have been executed by at least f+1 non-faulty replicas Achieved using a checkpoint, which occur when a request with sequence number (n) is divisible by some constant is executed 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 30 Wenbing Zhao

Garbage Collection n When a replica i produces a checkpoint it multicasts a message <CHECKPOINT, n, d, i> to other replicas Each replica collects checkpoint messages in its log until it has 2 f+1 of them for sequence number n with same digest d This creates a stable checkpoint and the replica discards all the pre-prepare, prepare and commit messages 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 31 Wenbing Zhao

View Changes n n Triggered by timeouts that prevent backups from waiting indefinitely for request to execute If the timer of backup expires in view v, the backup starts a view change to move to view v+1 by, q q Not accepting messages (other than checkpoint, viewchange, and new-view messages) Multicasting a VIEW-CHANGE message 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 32 Wenbing Zhao

View Changes n VIEW-CHANGE message is defined as <VIEW-CHANGE, v+1, n, C, P, i> where, C = 2 f + 1 checkpoint messages P = set of sets Pm Pm = a PRE-PREPARE msg + all PREPARE messages for all messages with committed = false 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 33 Wenbing Zhao

View Change - Primary n n Primary p of view v+1 receives 2 f valid VIEW-CHANGE messages Multicasts a <NEW-VIEW, v+ 1, V, O> message to all other replicas where q q n V = set of 2 f valid VIEW-CHANGE messages O = set of reissued PRE-PREPARE messages Moves to view v+1 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 34 Wenbing Zhao

View Changes - Backups n n Accepts NEW-VIEW by checking V and O Sends PREPARE messages for everything in O q n These PREPARE messages carry view v+1 Moves to view v+1 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 35 Wenbing Zhao

Events Before the View Change n n n Before the view change we have two groups of non-faulty replicas: the Confused minority and the Agreed majority A non-faulty replica becomes Confused when it is kept by the faulty's from agreeing on a sequence number for a request It can't process this request and so it will time out, causing the replica to vote for a new view 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 36 Wenbing Zhao

Events Before the View Change n n n The minority Confused replicas send a VIEW-CHANGE message and drop off the network The majority Agreed replicas continue working as long as the faulty's help with agreement The two groups can go out of synch but the majority keeps working until the faulty's cease helping with agreement 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 37 Wenbing Zhao

System State: Faulty Primary Is Erroneous View Change Possible? System State Confused Minority Agreed Majority ≤f non-faulty replicas ≥f+1 non-faulty replicas Agreed Majority ≥f+1 non-faulty replicas Confused Minority ≤f non-faulty replicas Adversary f non-faulty replicas f faulty replicas 10/2/2020 f non-faulty replicas f faulty replicas P P ≤ 2 f replicas: NOT enough to change views EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 38 Wenbing Zhao

Events Before the View Change n Given ≥f+1 non-faulty replicas that are trying to agree, the faulty replicas can either help that or hinder that ➲ ➲ n If they help, then agreement on request ordering is achieved and the clients get ≥f+1 matching replies for all requests with the faulty's help If they hinder, then the ≥f+1 non-faulty's will time out and demand for a new view When the new majority is in favor of a view change, we can proceed to the new view 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 39 Wenbing Zhao

System State: Faulty Primary Is it possible to continue processing requests? System State Confused Minority ≤f non-faulty replicas Agreed Majority ≥f+1 non-faulty replicas Adversary f non-faulty replicas f faulty replicas 10/2/2020 ≥f+1 non-faulty replicas Adversary P P f faulty replicas YES ≥ 2 f+1 replicas: enough for agreement EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 40 Wenbing Zhao

System State: Faulty Primary Majority now large enough to independently move to a new view Confused Minority ≤f non-faulty replicas Confused Majority 2 f+1 non-faulty replicas Enough to agree to change views Agreed Majority ≥f+1 non-faulty replicas Adversary P f non-faulty replicas f faulty replicas YES ≥ 2 f+1 replicas: enough for agreement Faulty replicas cease helping with agreement 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 41 Wenbing Zhao

Liveness n n Replicas must move to a new view if they are unable to execute a request To avoid starting a view change too soon, a replica that multicasts a view-change message for view v+1, waits for 2 f+1 view-change messages and then starts the timer T If the timer T expires before receiving new-view message it starts the view change for view v+2 The timer will wait 2 T before starting a view-change from v+2 to v+3 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 42 Wenbing Zhao

Liveness n n n If a replica receives f+1 valid view-change messages from other replicas for views greater than its current view, it sends a view-change message for the smallest view in the set, even if T has not expired Faulty replicas cannot cause a view-change by sending a view-change message since a viewchange will happen only if at least f+1 replicas send view-change message The above techniques guarantee liveness, unless message delays grow faster than the timeout period indefinitely 10/2/2020 EEC 688/788: Building Dependable Secure &Distributed Dependable Systems, Copyright Computing Wenbing Zhao 43 Wenbing Zhao

Zyzzyva: Speculative BFT n n A replica speculatively executes a request as soon as it receives a valid pre-prepare msg Commitment of a request is moved to the client q q n A request is said to have completed at hthe issuing client if the corresponding reply can be safely delivered If a request completes at a client, the request will eventually be committed at the server replicas Prepare and commit phases are reduced to a single phase q View change has one more additional phase Building Dependable Distributed Systems, Copyright Wenbing Zhao 44

Zyzzyva: Speculative BFT n n History hash: helps client determine if its request has been ordered appropriately Server replica maintains a history hash for each request ordered and appends the history hash hs = H(hs-1, ds) to the reply for the request with sequence number s q q ds: digest for the request Previous hash: hs-1, new hash with request s executed: hs H(): hash function Prefix concept: his is a prefix of hsj if sj > si, and there exist a set of requests with sequence numbers si+1, si+2, …, sj-1 with digests dsi+1, dsi+2, …, dsj-1 such that hsi+1=H(hsi, dsi+1), hsi+2 = H(hsi+1, dsi+2), …, hsj=H(hsj-1, dsj) Building Dependable Distributed Systems, Copyright Wenbing Zhao 45

Zyzzyva: Speculative BFT n n Safety: given any two requests that have completed, they must have been assigned two different sequence numbers. Furthermore, if the two sequence numbers are i and j an i < j, the history hash hi must be a prefix of hj Liveness: if a nonfaulty client issues a request, the request eventually completes Building Dependable Distributed Systems, Copyright Wenbing Zhao 46

Zyzzyva: Agreement Protocol n n A client maintains a complete timer for each request A request may complete at the client in one of two cases: q q Case 1: client receives 3 f+1 matching replies => all replicas have executed the request in the same total order Case 2: client receives at least 2 f+1 matching replies when timer expires. The client then initiates another round of msg exchange with the server replicas before the request is declared as complete Building Dependable Distributed Systems, Copyright Wenbing Zhao 47

Zyzzyva: Agreement Protocol, Case 1 Building Dependable Distributed Systems, Copyright Wenbing Zhao 48

Zyzzyva: Agreement Protocol, Case 2 Building Dependable Distributed Systems, Copyright Wenbing Zhao 49

Zyzzyva: Case 2 n Commit msg: contains a commit certificate: q q n Local-commit: a replica sends a local-commit when it receives a valid commit msg from client q n A list of 2 f+1 replica ids Signed component of spec-response from each of the 2 f+1 replicas Verify history hash When the client receives 2 f+1 or more local-commit, the request is completed and can be delivered Building Dependable Distributed Systems, Copyright Wenbing Zhao 50

Zyzzyva: View Change Protocol n When a view change is triggered q q n Sufficient number of backups timed out the current primary When the client receives spec-response msgs with different sequence numbers or different history hash values In the 2 nd case, client broadcasts a POM msg to all replicas q q Replica initiates a view change A replica also rebroadcasts POM upon receiving one => to speed up view change Building Dependable Distributed Systems, Copyright Wenbing Zhao 51

Zyzzyva: View Change Protocol n What is special in Zyzzyva q q n Impact on view change: q q n At most one round msg exchange during normal operation for agreement => equivalent to prepare phase (or commit phase), for case 2. Replicas would have commit certificate For case 1, replicas would not possess a commit certificate Need an additional round msg exchange – “I hate the primary” Need to change condition on including a request in new-view msg Cannot use PBFT view change protocol: may lose liveness q q q Faulty primary could force f nonfaulty replicas to suspect itself, but cooperate with other f+1 nonfaulty replicas f faulty replicas stops sending spec-response Client cannot complete any request: only receive f+1 matching replies Building Dependable Distributed Systems, Copyright Wenbing Zhao 52

Zyzzyva: View Change Protocol n PBFT view change q n A replica abandons the current view as soon as it suspects the primary: stops participating agreement for current view Zyzzyva view change q q Make sure a view change will actually take place before abandoning the current view Accomplished by using the “I hate primary” msg exchange: a replica abandons the current view only when it receives f+1 “I hate primary” msgs Building Dependable Distributed Systems, Copyright Wenbing Zhao 53

Zyzzyva: View Change Protocol n Zyzzyva view change: deal with case 1 q q A replica includes all order-req msgs received since latest stable checkpoint, or the most recent commit certificate New primary adopts request-seq# binding if f+1 or more matching order-req msgs The primary may see multiple sets for same seq# (for different requests): can select any of the bindings because none of them could have been completed A backup should accept the decision by the new primary for the ordering Building Dependable Distributed Systems, Copyright Wenbing Zhao 54