Fault Tolerance Chapter 7 Distributed Systems Mr Imran

Fault Tolerance Chapter – 7 (Distributed Systems) Mr. Imran Rao Ms. Niu. Yu 22 nd November 2005 Real Time Multimedia Lab

Today’s Agenda • • Overview Introduction to Fault Tolerance Process Resilience Reliable Client-Server communication Reliable group communication Distributed commit Recovery Summary Real Time Multimedia Lab

Objectives • After taking this presentation, one can – Define Fault Tolerance – Distinguish among different kinds of Faults – Apply Failure Masking Techniques • Process Resilience • Communication Failure Masking – What is Distributed Commit and Related Issues? – What are the Recovery Techniques? Real Time Multimedia Lab

Overview • Introduction to Fault Tolerance – Basic Concepts – Failure Modes – Failure Masking • Process Resilience – Design Issues • Reliable Communication – P 2 P Communication – Client Server Communication (RPC, RMI) – Group Communication (Multicasting) • Distributed Commit – Multi Phase Commit (Two & Three Phase) • Recovery Techniques – Check Pointing, Message Logging Real Time Multimedia Lab

Basic Concepts (1/3) • What is Failure? – System is said to be in failure state when it cannot meet its promise. • Why do Failure occurs? – Failures occurs because of the error state of the system. • What is the reason for Error? – The cause of an error is called a fault • Is there some thing ‘Partial Failure’? • Faults can be Prevented, Removed and Forecasted. • Can Faults be Tolerated by a system also? Real Time Multimedia Lab

Basic Concepts (2/3) • What characteristics makes a system Fault Tolerant? – – – Availability: System is ready to used immediately. Reliability: System can run continuously without failure. Safety: Nothing catastrophic happens if a system temporarily fails. Maintainability: How easy a failed system can be repaired. Dependability: ? ? ? • What is the reliability and availability of following systems? – If a system goes down for one millisecond every hour – If a System never crashes but is shut down for two weeks every August. Real Time Multimedia Lab

Basic Concepts (3/3) • Classification of Faults – Transient: Occurs once and than disappears – Intermittent: Occurs, vanishes on its own accord, than reappears and so on – Permanent: They occurs and doesn’t vanish until fixed manually. • Can you classify the Faults caused by following situations? – – A flying bird obstructing the transmitting waves signals A loosely connected power plug Burnt out chips Software Bugs • Which Fault you think is more difficult to detect and why? Real Time Multimedia Lab

Faults in Distributed Systems • If in a Distributed Systems some fault occurs, the error may by in any of – The collection of servers or – Communication Channel or – Even both • However, malfunctioning server itself may not always be the fault we are looking for. Why? • Dependency relations appear in abundance in DS. • Hence, we need to classify failures to know how serious a failure actually is. Real Time Multimedia Lab

Failure Models Type of Failure Description Crash Failure A server halts, but is working correctly until it halts Omission Failure • Receive omission • Send omission A server fails to respond to incoming requests A server fails to receive incoming messages A server fails to send messages Timing Failure A server's response lies outside the specified time interval Response Failure • Value • State Transition The server's response is incorrect The value of the response is wrong The server deviates from the correct flow of control Arbitrary Failure A server may produce arbitrary responses at arbitrary times Real Time Multimedia Lab

Failure Masking by Redundancy (1/3) • A system to be fault tolerant, the best it can do is try to hide the occurrence of failure from other processes • Key technique to masking faults is to use Redundancy. – Information redundancy: Extra bits are added to allow recovery from garbled bits – Time redundancy: An action is performed, and then, if need be, it is performed again. – Physical redundancy: Extra equipment or processes are added • Issue: How much redundancy is needed? Real Time Multimedia Lab

Failure Masking by Redundancy (2/3) • Some Examples of Redundancy Schemes – – – Hamming Code Transactions Replicated Processes or Components Aircraft has four engines, can fly with only three Sports game has extra referee. Mammals. Can you point out How? Real Time Multimedia Lab

Failure Masking by Redundancy (3/3) Figure: Fault Tolerance in Electronic Circuits • Triple modular redundancy: – If two or three of the input are the same, the output is equal to that input. – If all three inputs are different, the output is undefined. Real Time Multimedia Lab

What's Next • We have studied – – – What are Faults? What are the characteristics of a fault tolerant system? How to distinguish between different types of faults? What are the failure models? How Failures can be Masked? • What’s Next? – How Fault Tolerance is achieved in DS – How Process are made Resilient against Faults? Real Time Multimedia Lab

Process Resilience • Problem: – How fault tolerance in distributed system is achieved, especially against Process Failures? • Solution: – – Replicating processes into groups. Groups are analogous to Social Organizations. Consider collections of process as a single abstraction All members of the group receive the same message, if one process fails, the others can take over for it. – Process groups are dynamic and a Process can be member of several groups. – Hence we need some management scheme for groups. Real Time Multimedia Lab

Process Groups (1/2) Flat Group vs. Hierarchical Group • Flat Group • • • Advantage: Symmetrical and has no single point failure Disadvantage: Decision making is more complicated. Voting Hierarchical Group • • Advantage: Make decision without bothering others Disadvantage: Lost coordinator Entire group halts Real Time Multimedia Lab

Process Groups (2/2) Group Membership • Group Server (Client Server Model) – Straight forward, simple and easy to implement – Major disadvantage Single point of failure • Distributed Approach (P 2 P Model) – Broadcast message to join and leave the group – In case of fault, how to identify between a really dead and a dead slow member – Joining and Leaving must be synchronized on joining send all previous messages to the new member – Another issue is how to create a new group? Real Time Multimedia Lab

Failure Masking & Replication • • Replicate Process and organize them into groups Replace a single vulnerable process with the whole fault tolerant Group A system is said to be K fault tolerant if it can survive faults in K components and still meet its specifications. How much replication is needed to support K Fault Tolerance? – K+1 or 2 K+1 ? • Case: 1) If K processes stop, then the answer from the other one can be used. K+1 2) If meet Byzantine failure, the number is 2 K+1 Problem? Real Time Multimedia Lab

Agreement in Faulty Systems • Why we need Agreements? • Goal of Agreement – Make all the non-faulty processes reach consensus on some issue – Establish that consensus within a finite number of steps. • Problems of two cases – Good process, but unreliable communication • Example: Two-army problem – Good communication, but crashed process • Example: Byzantine generals problem Real Time Multimedia Lab

Two-army problem Red Troop 5000 1. Let us attack 3. I got your at 6 AM. Attack message. 3000 that is 4. 2. I Ok, knew that Attack yougood. got my message. 3000 Blue Troop Command by Alexander Command by Napoleon It is easy to show that Alexander and Napoleon will never reach agreement, no matter how many acknowledgements they send. (Due to unreliable communication). Real Time Multimedia Lab

Byzantine generals problem The Byzantine generals problem for 3 loyal generals and 1 traitor. a) The generals announce their troop strengths (in units of 1 thousand soldiers). b) The vectors that each general assembles based on (a) c) The vectors that each general receives in step 3. Real Time Multimedia Lab

Go forward one more step More than two-thirds agreement The same as in previous slide, except now with 2 loyal generals and one traitor. Lamport proved that in a system with m faulty processes, agreement can be achieved only if 2 m+1 correctly functioning processes are present, for a total of 3 m+1. Real Time Multimedia Lab

Reliable client-server communication What about reliable point-to-point transport protocols ? • TCP masks omission failures – … by using ACKs & retransmissions • … but it does not mask crash failures ! – E. g. : When a connection is broken, the client is only notified via an exception Real Time Multimedia Lab

Five classes of failures in RPC • Client is unable to locate server – Binding exception • … at the expense of transparency • Request message is lost – Is it safe to retransmit ? • Allow server to detect it is dealing with a retry • Server crashes after receiving a request • Reply message is lost • Client crashes after sending a request Real Time Multimedia Lab

Server Crashes (I) A server in client-server communication a) b) c) Normal case Crash after execution Crash before execution Real Time Multimedia Lab

Server Crashes (II) • At-least-once semantics – Client keeps retransmitting until it gets a response • At-most-once semantics – Give up immediately & report failure • Guarantee nothing Ideal would be exactly-once semantics • … no general way to arrange this ! Real Time Multimedia Lab

Server Crashes (III) • Print server scenario: – M: server’s completion message • Server may send M either before or after printing – P: server’s print operation – C: server’s crash • Possible event orderings: – – – M P C M C ( P) P M C P C ( M) C ( P M) C ( M P) Real Time Multimedia Lab

Server Crashes (IV) Client Server Strategy M -> P Strategy P -> M Reissue strategy MPC MC(P) C(MP) PMC PC(M) C(PM) Always DUP OK OK DUP OK Never OK ZERO OK OK ZERO Only when ACKed DUP OK ZERO Only when not ACKed OK ZERO OK OK DUP OK Different combinations of client & server strategies in the presence of server crashes. No combination of client & server strategy is correct for all cases ! Real Time Multimedia Lab

Lost Reply Messages • Is it safe to retransmit the request ? – Idempotent requests • Example: Read a file’s first 1024 bytes • Counterexample: money transfer order • Assign sequence number to request – Server keeps track of client’s most recently received sequence # – … additionally, set a RETRANSMISSION bit in the request header Real Time Multimedia Lab

Client Crashes (I) • Orphan computation: – No process waiting for the result • Waste of resources (CPU cycles, locks) • Possible confusion upon client’s recovery • 4 alternative strategies proposed by Nelson (1981) • Extermination: – Client keeps log of requests to be issued • Upon recovery, explicitly kill orphans – Overhead of logging (for every RPC) – Problems with grand-orphans – Problems with network partitions Real Time Multimedia Lab

Client Crashes (II) • Reincarnation: – Divide time up into epochs (sequentially numbered) – Upon reboot, client broadcasts start-of-epoch • Upon receipt, all remote computations on behalf of this client are killed • After a network partition, an orphan’s response will contain an obsolete epoch number easily detected • Gentle reincarnation: – Upon receipt of start-of-epoch, each server checks to see if it has any remote computations • If the owner cannot be found, the computation is killed • Expiration: – Each RPC is given a time quantum T to complete • … must explicitly ask for another if it cannot finish in time • After reboot, client only needs to wait a time T … • How to select a reasonable value for T ? Real Time Multimedia Lab

Agenda Introduction to Fault Tolerance Process Resilience Reliable Client-Server communication Reliable group communication Distributed commit Recovery Summary Real Time Multimedia Lab

Basic Reliable-Multicasting Schemes a) b) A simple solution to reliable multicasting when all receivers are known & are assumed not to fail Message transmission Reporting feedback Real Time Multimedia Lab

Scalability in Reliable Multicasting • The scheme described above can not support large numbers of receivers. • Reason: Feedback Implosion Receivers are spread across a wide-area network • Solution: Reduce the number of feedback messages that are returned to the sender. • Model: Feedback suppression Real Time Multimedia Lab

Nonhierarchical Feedback Control Several receivers have scheduled a request for retransmission, but the first retransmission request leads to the suppression of others. Real Time Multimedia Lab

Hierarchical Feedback Control a) b) The essence of hierarchical reliable multicasting: Each coordinator forwards the message to its children. A coordinator handles retransmission requests. Real Time Multimedia Lab

Atomic Multicast • We need to achieve reliable multicasting in the presence of process failures. • Atomic multicast problem: a message is delivered to either all processors or to none at all messages are delivered in the same order to all processes • Virtually synchronous reliable multicasting offering totallyordered delivery of messages is called atomic multicasting Real Time Multimedia Lab

Virtual Synchrony (I) The logical organization of a distributed system to distinguish between message receipt and message delivery Real Time Multimedia Lab

Virtual Synchrony (II) • Reliable multicast guarantees that a message multicast to group view G is delivered to each nonfaulty process in G. • If the sender of the message crashes during the multicast, the message may either be delivered to all remaining processes, or ignored by each of them. • A reliable multicast with this property is said to be virtually synchronous • All multicasts take place between view changes. A view change acts as a barrier across which no multicast can pass Real Time Multimedia Lab

Virtual Synchrony (III) The principle of virtual synchronous multicast. Real Time Multimedia Lab

Implementing Virtual Synchrony a) b) c) Process 4 notices that process 7 has crashed, sends a view change Process 6 sends out all its unstable messages, followed by a flush message Process 6 installs the new view when it has received a flush message from everyone else Real Time Multimedia Lab

Message Ordering (I) 1. 2. Unordered multicast Process P 1 Process P 2 Process P 3 sends m 1 receives m 2 sends m 2 receives m 1 FIFO-ordered multicast Process P 1 Process P 2 Process P 3 Process P 4 sends m 1 receives m 3 sends m 2 receives m 3 receives m 1 sends m 4 receives m 2 receives m 4 Real Time Multimedia Lab

Message Ordering (II) 3. 4. Reliable causally-ordered multicast delivers messages so that potential causality between different messages is preserved Total-ordered delivery Multicast Basic Message Ordering Total-ordered Delivery? Reliable multicast None No FIFO multicast FIFO-ordered delivery No Causal multicast Causal-ordered delivery No Atomic multicast None Yes FIFO atomic multicast FIFO-ordered delivery Yes Causal atomic multicast Causal-ordered delivery Yes Real Time Multimedia Lab

Agenda Introduction to Fault Tolerance Process Resilience Reliable Client-Server communication Reliable group communication Distributed commit Recovery Summary Real Time Multimedia Lab

Two-phase Commit (I) a) b) The finite state machine for the coordinator in 2 PC. The finite state machine for a participant. • Process crashes other processes may be indefinite waiting for a message This protocol can easily fail • timeout mechanisms are used Real Time Multimedia Lab

Failure handling in 2 PC • Participant times out waiting for coordinator’s Request-to -prepare – It decide to abort. • Coordinator times out waiting for a participant’s vote – It decides to abort. • A participant that voted Prepared times out waiting for the coordinator’s decision – It’s blocked. – Use a termination protocol to decide what to do. – Native termination protocol – wait until coordinator recovers. • The coordinator times out waiting for ACK message – It must resolicit them, so it can forget the decision Real Time Multimedia Lab

Actions by coordinator while START _2 PC to local log; multicast VOTE_REQUEST to all participants; while not all votes have been collected { wait for any incoming vote; if timeout { while GLOBAL_ABORT to local log; multicast GLOBAL_ABORT to all participants; exit; } record vote; } if all participants sent VOTE_COMMIT and coordinator votes COMMIT{ write GLOBAL_COMMIT to local log; multicast GLOBAL_COMMIT to all participants; } else { write GLOBAL_ABORT to local log; multicast GLOBAL_ABORT to all participants; } Real Time Multimedia Lab

Actions by participant write INIT to local log; wait for VOTE_REQUEST from coordinator; if timeout { write VOTE_ABORT to local log; exit; } if participant votes COMMIT { write VOTE_COMMIT to local log; send VOTE_COMMIT to coordinator; wait for DECISION from coordinator; if timeout { multicast DECISION_REQUEST to other participants; wait until DECISION is received; /* remain blocked */ write DECISION to local log; } if DECISION == GLOBAL_COMMIT write GLOBAL_COMMIT to local log; else if DECISION == GLOBAL_ABORT write GLOBAL_ABORT to local log; } else { write VOTE_ABORT to local log; send VOTE ABORT to coordinator; } Real Time Multimedia Lab

3 Phase Commit Protocol (I) • Problem of 2 PC – In some cases, participants cannot reach a final decision – What case? • Solution 1. No single state from which to make a transition to either COMMIT or ABORT. 2. No state in which not possible to make a final decision and from which a transition to COMMIT can be made Real Time Multimedia Lab

Three-phase Commit Protocol (II) • Introduction of another Phase a) b) Finite state machine for the coordinator in 3 PC Finite state machine for a participant Real Time Multimedia Lab

Failure handling in 3 PC (I) • Coordinator Timeout/Recovery – Coordinator • WAIT: timeout – > ABORT • PRECOMMIT: timeout – > COMMIT – Recovering participant? Real Time Multimedia Lab

Failure handling in 3 PC (II) • Participant Timeout/Recovery – Participant • INIT: timeout -> ABORT • READY/PRECOMMIT: timeout (coordinator crashed) -> Ask other participants – Any ABORT/COMMIT: follow it – Any INIT: ABORT (no PRECOMMIT possible? ) – All(majority) READY: -> ABORT » Recovering participant = INIT: -> ABORT = READY: contact other participants = ABORT = PRECOMMIT: -> ABORT (cf. 2 PL) – All(majority) PRECOMMIT: COMMIT » Recovering participant = READY = recover to: READY/PRECOMMIT/COMMIT Real Time Multimedia Lab

Agenda Introduction to Fault Tolerance Process Resilience Reliable Client-Server communication Reliable group communication Distributed commit Recovery Summary Real Time Multimedia Lab

Recovery-Introduction q Goal: replace an erroneous state with an error-free state q Backward recovery: to restore such a recorded state when things go wrong q Combining checkpoints and message logging q Forward recovery: an attempt is made to bring the system in a correct new state from which it can continue to execute Real Time Multimedia Lab

Stable Storage a) b) c) Stable Storage Crash after drive 1 is updated Bad spot Stable storage is well suited to applications that require a high degree of fault tolerance Real Time Multimedia Lab

Checkpointing A recovery line: the most recent distributed snapshot Real Time Multimedia Lab

Independent Checkpointing The domino effect. Real Time Multimedia Lab

Message Logging Incorrect replay of messages after recovery, leading to an orphan process. Real Time Multimedia Lab

Summarization (I) q Fault tolerance is defined as the characteristic by which a system can mask the occurrence and recovery from failures q There exist several types of failures : Crash failure, Omission failure, Timing failure, Response failure, Arbitrary/Byzantine failure q Redundancy is the key technique needed to achieve fault tolerance q Reliable group communication is suitable for small groups Real Time Multimedia Lab

Summarization (II) q Atomic multicasting can be precisely formulated in terms of a virtual synchronous execution model q Group membership change agreement on the same list of members using commit protocol q Recovery in fault-tolerant systems is invariably achieved by checkpointing with message logging Real Time Multimedia Lab

QUESTIONS & ANSWERS Real Time Multimedia Lab