Operating System Reliability Andy Wang COP 5611 Advanced

Operating System Reliability Andy Wang COP 5611 Advanced Operating Systems

Some Axioms n n n Some simple systems, designed from scratch, sometimes work A complex system that works is invariably found to have evolved from a simple system that works A complex system, designed from scratch never works

Failure-Mode Theorems n n n Complex system usually operate in failure mode A system should have safe behaviors when encountering failures When a “fail-safe” system fails, it fails by failing to fail safe

Some definitions n Failure of a system occurs when the system does not perform its services in the manner specified q n Fault is anomalous physical condition q n Sometimes failures are subtle (e. g. , performance fault) Includes system specification/implementation mistakes Error is part of system state that differs from its intended value

Classification of Failures n n Process failures System failures Secondary storage failures Communication medium failures

Process Failures n Examples q q q n Errors leading to failure q q n Computation results in incorrect outcome System state deviates from specification Process fails to progress Deadlock, timeout, protection violation Bad input, consistency violation Ignoring malicious behavior

System Failures n Processor fails to execute q n n Software error, hardware error (CPU, bus, etc. ) Fail-stop behavior assumed Failure types q q Amnesia Partial-amnesia Pause Halting

Secondary Storage Failures n Stored data inaccessible q q q n n Parity error Head crash Contaminated medium Reconstructable from archive + log, maybe Mirrored disks (independent failure mode)

Communication Medium Failures n n Site can’t communicate with another site Causes q Switching node failure n n n q Link failure n q n Hardware failure Software failure Congestion Hardware Implementation failure Network partitions can result

Recovery n n Restart process/processor Reclaim resources Undo/finish incomplete transactions Concurrency makes things harder

Forward Error Recovery n n Goal: To restore system from erroneous state to error-free state If nature of error is completely known q q n Remove error from state Proceed with execution from error-free state Rarely possible to do

Backward Error Recovery n When error source unknown q q q Restore state to previous error-free state; restart Independent of fault, errors causing fault Problems n n Performance penalty No guarantee fault will not reoccur Possible unrecoverable component of state Recovery point: state used to replace error

Backward Error Recovery n Basic approaches q Operation-based n Logs q q q Update-in-place Write-ahead-log State-based

Update-in-Place n Every update to object also records the log q q n Name of object Old and new states of object Recoverable update operation implements as q Do, undo, redo operations

Write-ahead Log n n Update-in-place has problem if crash occurs between update and log recorded to stable storage Update object only after undo log recorded Before committing updates, record both redo and undo logs Expensive to write log to stable storage

State-Based Recovery n Save entire process state at recovery point q q q n Recovery point called checkpoint Rolling back process: restoring to checkpoint Tradeoff: frequent checkpoints vs. completion delay Shadow pages q q Save unmodified page copy on stable storage Update only volatile copy; discard on rollback

Concurrent Systems Recovery n Rollback issues q q Orphan messages Domino effect Lost messages Livelocks

Orphan Messages X x 1 [ y 1 [ Y Z x 2 [ m z 1 [ [ z 2 [ [ y 2 recovery point

Domino Effect n Suppose Y rolls back to y 2 q q n m is orphan message Process Y must rollback to y 1 Suppose Z rolls back to z 2 q q Y rolls back to y 1 Forcing Z to roll back to z 1

Lost Messages x 1 [ X m z 1 [ Z failure [ recovery point

Live Locks X x 1 [ Z z 1 [ [ repeated failure recovery point

Concurrent Recovery n n Coordination required at either time of establishing checkpoints Beginning of recovery

Checkpoint Assumptions n n Communication via messages Unreliable FIFO channels q q n Higher-level end-to-end protocols assumed Subsumes rollback-caused message loss No network partitions from communication failures

Checkpoint Algorithm Concepts n Permanent and tentative checkpoints q q Saved on stable storage Permanent: part of known consistent global checkpoint Tentative: until successful termination of checkpoint algorithm Rolls back only to permanent checkpoints

Synchronous Checkpoint Algorithms n n Two-phase commit Problems: q q q Message overhead for synchronizations Synchronization delays Costly when failures are rare

Asynchronous Checkpointing n n Local checkpoints taken independently Log all incoming messages on stable storage q q Minimizes undone computation Allows reprocessing of messages after rollback

Asynchronous Checkpointing Assumptions n Assumptions q q q Reliable FIFO communication channels Infinite buffers Event-driven computation n n A process idle until message received Processes message and change state Sends zero or more messages Can identify each event with monotonically increasing counter

Event-Driven Computation x 1 x 2 X y 1 y 2 Y z 1 Z z 2

Asynchronous Checkpointing n Basic idea q q Save states, messages sent at each event Volatile logging Each processor notes number of messages sent to others, and received from others Use counters to determine orphan messages

Summary n n n Failures caused by errors Can remove errors by forward/backward error recovery Backward error-recovery more costly, more general Synchronous checkpoints helpful, costly Asynchronous checkpoints messier, domino effects