Fault Tolerance Fault tolerance terminology n dependability extent

Fault Tolerance

Fault tolerance terminology n “dependability” - extent to which reliance can justifiably be placed on service. u General concept n “reliability” - continuity of service u metric: mean time between failures (MBTF) n “availability” - readiness for usage n “safety” - avoidance of catastrophic effects on environment n “security” - resistance to unauthorized access. 2

Faults, errors, failures n “fault” - component malfunction n “error” - system state is wrong n “failure” - system departs from specification fault error failure 3

System components fault failure Environment 4

Coping with faults n Reduce/eliminate faults in components. n Fault tolerance u. Prevent faults from becoming failures uusually through redundancy. 5

Types of faults (fault models) Fault tolerance algorithms dependent on fault models. n “Crash fault” or “stop fault” - faulty component stops responding. No incorrect state changes in component. n “Timing fault” - response is too early or late. n “Byzantine fault” - arbitrary behavior. Can be considered adversarial (imagine worst case). 6

The agreement problem n Processors may fail n … so, use multiple processors n … but then, processors may disagree, causing failures. n Need a principled approach to distributed agreement 7

Example: AFTI 16 (from J. Rushby) n “Advanced Fighter Technology Integration F 16 n Triple-redundant digital flight-control system (DFCS) with analog backup n DFCS design was “asynchronous” u processors ran independently sample sensor, evaluate control law, send command to actuator averages or selects from commands o u General Dynamics felt synchronization would introduce a single point of failure. 8

AFTI 16 problems n Processors can get widely varying sensor readings because of timing differences n Reconfiguration cause sudden changes in control (“thumps”). u. Need to allow wide range of “plausible values” before declaring a processor “bad” u. Bad sensor reading drags average down u. Sensor finally crosses threshhold and is called “bad” uaverage suddenly snaps back when sensor is excluded. 9

AFTI 16 problems (cont) n Processor states can diverge rapidly uespecially when different processors go into different control modes. n Design complexity u 70% of application code was for redundancy management u. Control laws had to be modified to ramp changes in and out smoothly 10

AFTI 16 flight test, Flight 36 n “Departure” from control laws for 3 seconds n acceleration exceeded -4 g, then +7 g n Angle of attack went to -10 degrees, then +20 degrees n Aircraft rolled 360 degreees n Cause: side air probe cut out at high angle of attack n Analysis showed this would cause complete failure of DFCS for several areas of flight envelope 11

AFTI 16 flight 44 n Each channel declared the others failed uasynchronous noise operation, timing skew, sensor n analog backup not selected usimultaneous anticipated failure of two channels not n Aircraft flown home on a single digital channel (not designed for this) n There were no hardware failures. 12

AFTI 16 Analysis (NASA) n Nearly all failure indications were design oversights related to asynchronous operation n Failures due to lack of understanding of interactions among u. Air data system uredundancy management software uflight control laws (decision points, thumps, ramp-in/out) n Moral of the story: Reliability through redundancy is a lot harder than it looks. 13

Distributed consensus n Goal: multiple processors agree on something in the presence of various kinds of faults and errors n Intellectually difficult u. Algorithms are tricky u. Proofs are subtle u. Sensitive to assumptions o Synchronous vs. asynchronous o Communication mechanism o Fault models n Many papers written 14

Synchronous vs. asynchronous n Synchronous: Processors run in lock-step u. Hard to implement - model may be unrealistic o Requires u. Consensus clock synchronization. is easier n Asynchronous: Processors run at arbitrary speed u. Easier to implement - model is conservative u. In most models, consensus problem is provably unsolvable. 15

Synchronous vs. asynchronous n Semi-synchronous u. Bounds on how far out-of-sync processors can get u. Model is fairly realistic u. Consensus is almost as easy as synchronous 16

Fault models n Goal: Make claims such as: “the system will continue to function if any single processor stops. ” n More conservative fault models: u. Fault tolerance is harder u. But, if successful, stronger claims can be made u. Fewer assumptions = simpler FMEA, easier “certification” n A lot of models have been proposed. 17

Process fault models n “Stopping fault” - process stops sending messages udoes not restart udoes not send wrong messages uliberal (easy) model n “Byzantine fault” - process behaves arbitrarily u. Name comes from cute “Byzantine generals” metaphor u. May send arbitrary messages, enter arbitrary states u. Equivalent to “evil” behavior, for our purposes 18

Synchronous agreement with stopping faults n Multiple processes want to “agree” on a value n Applications usensor readings among redundant processors udecide what time it is udecide which of a group of processors are broken and should be removed from system. 19

Synchronous agreement - properties n Each process starts with an initial value, processes end with a decision value. n Agreement: all good processes decide on same values. n Validity: if all processors start with same value, that value is the final decision value. n Termination: All good processes eventually decide. 20

Flood set algorithm n Assumption: There is a dedicated link between each pair of processes n No more than f processes can stop n Each process has an initial value v n Each process accumulates a set W of all the values it has ever seen. u. On each round, every process sends its W set to every other process u. Every process sets W to the union of the old value and all the new values coming in from others. 21

Flood set n After f rounds, every process looks at W. u. If W has only one value, choose that value. u. Else, choose 0 (a predetermined default). 22

Flood set correctness n In f+1 rounds, there must be at least one round in which no processes stop u. At most f processes can stop, and processes cannot stop more than once. n If no process stops in round r, W will be the same in all good processes in subsequent rounds. u. All good processes successfully send all values in W to all other good processes, so all processes will have same W after the round. u. After this, nothing can get added to any W sets, so it doesn’t matter whether more 23 processes stop.

Flood set correctness n So, after f+1 rounds, all non-stopped processes have same W sets u. If W has only one value, all processes pick this value. u. Else all processes pick 1. 24

Flood set example n 3 processes, 1 fault, default value = 0 something Dies after A A B W in round 0 {A} {B} W in round 1 {A, B} {A} - sending W to something but not {A, B} - 0 0 - s W in round 2 something W sets for , are same Www blank here Choose default blank here because |W|>1 Blank here final 25

Flood set efficiency O((f + 1) n 2) messages f+1 rounds n processes send n messages per round O((f+1)n 3) values are sent (each message may have a set of up to n values) 26

Optimized flood set n Note: If W has more than one element, process doesn’t need to know what is in it. n Idea: Every process sends only first two distinct values. u Every process sends its initial value on first round u If process receives a different value, it sends it out on next round n Correctness proof: run Flood and Opt. Flood in parallel u same initial values, stopping pattern u W sets have more than one value iff Opt. Flood process gets two values. 27

Opt. Flood efficiency 2 n 2 messages n processes send at most two messages to n other processes. O(n 2) values are sent 28

Byzantine agreement n Goal: non-faulty processes should agree on a value. u E. g. , message received u e. g. , sensor value n Faults may cause arbitrary behavior u arbitrary values communicated u different values communicated to different receivers n Advantage: reduces fault analysis n Disadvantage: hard or impossible to do. 29

Byzantine agreement properties Agreement: All good processes agree on a value Validity: If source of value was non-faulty, agreed upon value is the same. 30

Asynchronous agreement n Asynchronous model: u Message transmission takes arbitrary time. u Processes run at arbitrary speeds. n Theorem: There is no algorithm that reaches agreement in an asynchronous model with even one Byzantine failure u Fine print: Details of conditions, communication n This is one of the most important results about distributed systems. 31

Synchronous agreement n Synchronous model: Processes can communicate in a sequence of rounds. All processes complete a round before next round begins. n The agreement problem is solvable in this model. n Theorem: Tolerating k Byzantine faults requires > 3 k processes. n So “Triple modular redundancy” can’t handle Byzantine faults. n Practical case: 1 Byzantine fault, 4 processes. n Assumes full connectivity (connections between each pair of processors). 32

Synchronous agreement with one fault n Single transmitter communicates value to all processes. n Round 0: Transmitter sends value to n-1 receivers. u Values are sent correctly if transmitter is not faulty. n Round 1: Each receiver sends value to n-2 other receivers. u Receivers record all values separately. u Intuition: receivers compare notes on what transmitter told them. n Each receiver choose majority value of all values it received. u If no majority, use pre-arranged default value. 33

Example 1 - faulty transmitter Round 0: faulty xmtr sends P 1 varying results to rcvrs. 1 P 2 1 P 3 2 Xmtr R c v r P 1 P 2 P 3 consensus P 1 1 1 2 1 P 2 1 1 2 1 P 3 1 1 2 1 take majority of all Round 1: rcvrs exchange values (reliably) Finally, receivers answers These are the round 0 values 34

Example 2 - faulty transmitter Round 0: faulty xmtr sends P 1 varying results to rcvrs. 1 P 2 2 P 3 3 Xmtr R c v r P 1 P 2 P 3 consensus P 1 1 2 3 0 P 2 1 2 3 0 P 3 1 2 3 0 Round 1: rcvrs exchange values (reliably) There is no majority, so rcvrs use default These are the round 0 values 35

Example 3 - faulty receiver Round 0: faulty xmtr sends P 1 varying results to rcvrs. 1 P 2 1 P 3 1 Process 1 is Xmtr R c v r P 1 P 2 P 3 consensus P 1 1 5 P 2 2 1 1 1 P 3 3 1 1 1 broken, so result is not required to be correct Majority computes correct values for Process 1 sends bogus values processes 2, 3 These are the round 0 values 36

General case n Previous algorithm can be generalized to handle more Byzantine faults. n General results: k faults require k+1 (k? ) rounds, 3 k+1 processors n Number of messages grows exponentially with number of rounds n Intuition: “Pn said that Pn-1 said that. . . p 1 said that p 0 said that the value was x” u There are exponentially many chains pn. . . p 0. 37

Hybrid Byzantine agreement n Idea: Free bonus reliability with the purchase of Byzantine agreement. n Handles Byzantine faults, plus some more simpler faults n Symmetric fault: process sends same wrong value to everyone. n Nonmalicious fault: process sends a recognizable error value. n Advantages: u If processors have these faults, we can tolerate more faulty processors u These faults are more probable than true Byzantine faults - so this increases reliability 38

Hybrid Byzantine agreement n Modify previous algorithm by adding special error value “E”. u Nonmalicious faults send E value (other faults may send E, also). u Majority algorithm first removes E values. n Theorem: Algorithm reaches agreement if n n > 2 a + 2 s + b + r ua = Byzantine, s = symmetric, b = nonmalicious, r = number of rounds (excluding first transmission). u Previous case: a=1, s=0, b=0, r=1, so n > 3 u With 6 processors, can deal with 1 Byzantine + 2 nonmalicious faults. u or 1 Byzantine and 1 symmetric u. . . but just 1 Byzantine in previous algorithm 39

Variations n Synchronous communication is difficult u Compromise between synchronous and asynchronous: real-time constraints. n “Authentication” - agreement can be made less costly by using digital signatures u transmitter digitally signs messages u processes can’t lie about who said what. u can handle any number of faults (in synchronous model). n May assume different network connectivity u Some links in network missing 40

Summary n Fault tolerance is tricky. Redundancy does not necessarily buy reliability. n Byzantine models can account for unforeseen fault types. n Byzantine agreement is impossible in some models. n There exist practical algorithms for Byzantine agreement if synchronous communication is available. n There are deep theoretical results in this area. 41