CENG 334 Introduction to Operating Systems Deadlocks Topics

  • Slides: 44
Download presentation
CENG 334 Introduction to Operating Systems Deadlocks Topics: Deadlocks • Dining philosopher problem •

CENG 334 Introduction to Operating Systems Deadlocks Topics: Deadlocks • Dining philosopher problem • Erol Sahin Dept of Computer Eng. Middle East Technical University Ankara, TURKEY URL: http: //kovan. ceng. metu. edu. tr/~erol/Courses/ceng 334 1

What’s a deadlock? 2

What’s a deadlock? 2

Deadlock A deadlock happens when Two (or more) threads waiting for each other None

Deadlock A deadlock happens when Two (or more) threads waiting for each other None of the deadlocked threads ever make progress Mutex 1 holds Thread 1 waits for Mutex 2 Adapted from Matt Welsh’s (Harvard University) slides. holds Thread 2 3

Deadlock Definition Two kinds of resources: Preemptible: Can take away from a thread e.

Deadlock Definition Two kinds of resources: Preemptible: Can take away from a thread e. g. , the CPU Non-preemptible: Can't take away from a thread e. g. , mutex, lock, virtual memory region, etc. Why isn't it safe to forcibly take a lock away from a thread? Starvation A thread never makes progress because other threads are using a resource it needs Deadlock A circular waiting for resources Thread A waits for Thread B waits for Thread A Starvation ≠ Deadlock Adapted from Matt Welsh’s (Harvard University) slides. 4

Dining Philosophers Classic deadlock problem Multiple philosophers trying to lunch One chopstick to left

Dining Philosophers Classic deadlock problem Multiple philosophers trying to lunch One chopstick to left and right of each philosopher Each one needs two chopsticks to eat Adapted from Matt Welsh’s (Harvard University) slides. 5

Dining Philosophers What happens if everyone grabs the chopstick to their right? Everyone gets

Dining Philosophers What happens if everyone grabs the chopstick to their right? Everyone gets one chopstick and waits forever for the on the left All of the philosophers starve!!! Adapted from Matt Welsh’s (Harvard University) slides. 6

Deadlock Characterization Deadlock can arise if four conditions hold simultaneously. Mutual exclusion: only one

Deadlock Characterization Deadlock can arise if four conditions hold simultaneously. Mutual exclusion: only one process at a time can use a resource. Hold and wait: a process holding at least one resource is waiting to acquire additional resources held by other processes. No preemption: a resource can be released only voluntarily by the process holding it, after that process has completed its task. Circular wait: there exists a set {P 0, P 1, …, P 0} of waiting processes such that • P 0 is waiting for a resource that is held by P 1, • P 1 is waiting for a resource that is held by P 2, …, • Pn– 1 is waiting for a resource that is held by Pn, and • P 0 is waiting for a resource that is held by P 0. Adapted from Operating System Concepts (Silberschatz, Galvin, Gagne) slides. 7

Deadlock Prevention Restrain the ways request can be made to ensure that at least

Deadlock Prevention Restrain the ways request can be made to ensure that at least one of the four conditions DO NOT HOLD! Mutual Exclusion • not required for sharable resources; • must hold for non-sharable resources, • such as a printer. Hold and Wait • • • must guarantee that whenever a process requests a resource, it does not hold any other resources. Require process to request and be allocated all its resources before it begins execution, or allow process to request resources only when the process has none. • low resource utilization; • starvation possible. 8

Deadlock Prevention (Cont. ) No Preemption • • If a process that is holding

Deadlock Prevention (Cont. ) No Preemption • • If a process that is holding some resources requests another resource that cannot be immediately allocated to it, then all resources currently being held are released. Preempted resources are added to the list of resources for which the process is waiting. Process will be restarted only when it can regain its old resources, as well as the new ones that it is requesting. Can be applied to resources whose state can be saved such as CPU, and memory. Not applicable to resources such as printer and tape drives. Circular Wait • impose a total ordering of all resource types, and • require that each process requests resources in an increasing order of enumeration. 9

Circular Wait - 1 Each resource is given an ordering: F(tape drive) = 1

Circular Wait - 1 Each resource is given an ordering: F(tape drive) = 1 F(disk drive) = 2 F(printer) = 3 F(mutex 1) = 4 F(mutex 2) = 5 ……. Each process can request resources only in increasing order of enumeration. A process which decides to request an instance of Rj should first release all of its resources that are F(Ri) >= F(Rj). 10

Circular Wait - 2 For instance an application program may use ordering among all

Circular Wait - 2 For instance an application program may use ordering among all of its synchronization primitives: F(semaphore 1) = 1 F(semaphore 2) = 2 F(semaphore 3) = 3 ……. After this, all requests to synchronization primitives should be made only in the increasing order: Correct use: down(semaphore 1); down(semaphore 2); Incorrect use: down(semaphore 3); down(semaphore 2); Keep in mind that it’s the application programmer’s responsibility to obey this order. 11

Methods for Handling Deadlocks How should we handle deadlocks Ensure that the system will

Methods for Handling Deadlocks How should we handle deadlocks Ensure that the system will never enter a deadlock state. Allow the system to enter a deadlock state and then recover. Ignore the problem and pretend that deadlocks never occur in the system; used by most operating systems, including UNIX. Adapted from Operating System Concepts (Silberschatz, Galvin, Gagne) slides. 12

Dining Philosophers How do we solve this problem? ? (Apart from letting them eat

Dining Philosophers How do we solve this problem? ? (Apart from letting them eat with forks. ) Adapted from Matt Welsh’s (Harvard University) slides. 13

How to solve this problem? Solution 1: Don't wait for chopsticks Grab the chopstick

How to solve this problem? Solution 1: Don't wait for chopsticks Grab the chopstick on your right Try to grab chopstick on your left If you can't grab it, put the other one back down Breaks “no preemption” condition – no waiting! Solution 2: Grab both chopsticks at once Requires some kind of extra synchronization to make it atomic Breaks “multiple independent requests” condition! Solution 3: Grab chopsticks in a globally defined order Number chopsticks 0, 1, 2, 3, 4 Grab lower-numbered chopstick first Means one person grabs left hand rather than right hand first! Breaks “circular dependency” condition Solution 4: Detect the deadlock condition and break out of it Scan the waiting graph and look for cycles Shoot one of the threads to break the cycle Adapted from Matt Welsh’s (Harvard University) slides. 14

Deadlock Avoidance Requires that the system has some additional a priori information available. •

Deadlock Avoidance Requires that the system has some additional a priori information available. • Simplest and most useful model requires that each process declare the maximum number of resources of each type that it may need. • • The deadlock-avoidance algorithm dynamically examines the resource-allocation state to ensure that there can never be a circular-wait condition. • • Is this possible at all? When should the algorithm be called? Resource-allocation state is defined by the number of available and allocated resources, and the maximum demands of the processes. 15

System Model Resource types R 1, R 2, . . . , Rm CPU,

System Model Resource types R 1, R 2, . . . , Rm CPU, memory, I/O devices disk network Each resource type Ri has Wi instances. For instance a quad-core processor has 4 CPUs Each process utilizes a resource as follows: request use release Adapted from Operating System Concepts (Silberschatz, Galvin, Gagne) slides. 16

Resource-Allocation Graph A set of vertices V and a set of edges E. V

Resource-Allocation Graph A set of vertices V and a set of edges E. V is partitioned into two types: P = {P 1, P 2, …, Pn}, the set consisting of all the processes in the system. R = {R 1, R 2, …, Rm}, the set consisting of all resource types in the system. request edge – directed edge P 1 Rj assignment edge – directed edge Rj Pi Adapted from Operating System Concepts (Silberschatz, Galvin, Gagne) slides. 17

Resource Allocation Graph With A Deadlock If there is a deadlock => there is

Resource Allocation Graph With A Deadlock If there is a deadlock => there is a cycle in the graph. However the reverse is not true! i. e. If there is a cycle in the graph =/> there is a deadlock Adapted from Operating System Concepts (Silberschatz, Galvin, Gagne) slides. 18

Resource Allocation Graph With A Cycle But No Deadlock However the existence of a

Resource Allocation Graph With A Cycle But No Deadlock However the existence of a cycle in the graph does not necessarily imply a deadlock. Overall message: If graph contains no cycles no deadlock. If graph contains a cycle if only one instance per resource type, then deadlock. if several instances per resource type, possibility of deadlock. Adapted from Operating System Concepts (Silberschatz, Galvin, Gagne) slides. 19

Resource-Allocation Graph Algorithm Claim edge Pi Rj indicated that process Pj may request resource

Resource-Allocation Graph Algorithm Claim edge Pi Rj indicated that process Pj may request resource Rj; represented by a dashed line. Claim edge converts to request edge when a process requests a resource. When a resource is released by a process, assignment edge reconverts to a claim edge. Adapted from Operating System Concepts (Silberschatz, Galvin, Gagne) slides. 20

Resource-Allocation Graph Algorithm Claim edge Pi Rj indicated that process Pj may request resource

Resource-Allocation Graph Algorithm Claim edge Pi Rj indicated that process Pj may request resource Rj; represented by a dashed line. Claim edge converts to request edge when a process requests a resource. When a resource is released by a process, assignment edge reconverts to a claim edge. Cycle => Unsafe Resources must be claimed a priori in the system. Note that the cycle detection algorithm does not work with resources that have multiple instances. Adapted from Operating System Concepts (Silberschatz, Galvin, Gagne) slides. 21

Safe, unsafe and deadlock states If a system is in safe state no deadlocks.

Safe, unsafe and deadlock states If a system is in safe state no deadlocks. If a system is in unsafe state possibility of deadlock. Avoidance ensure that a system will never enter an unsafe state. Adapted from Operating System Concepts (Silberschatz, Galvin, Gagne) slides. 22

Safe State When a process requests an available resource, system must decide if immediate

Safe State When a process requests an available resource, system must decide if immediate allocation leaves the system in a safe state. System is in safe state if there exists a safe sequence of all processes. Sequence

is safe if for each Pi, the resources that Pi can still request can be satisfied by currently available resources + resources held by all the Pj, with j < i. If Pi resource needs are not immediately available, then Pi can wait until all Pj have finished. When Pj is finished, Pi can obtain needed resources, execute, return allocated resources, and terminate. When Pi terminates, Pi+1 can obtain its needed resources, and so on. Adapted from Operating System Concepts (Silberschatz, Galvin, Gagne) slides. 23

Banker’s Algorithm While giving credits, a banker should ensure that it never allocates all

Banker’s Algorithm While giving credits, a banker should ensure that it never allocates all of its cash in such a way that none of its creditors can finish their work and pay back the loan. 24

Example The system has three processes and 12 tape drives. t=t 0 P 1

Example The system has three processes and 12 tape drives. t=t 0 P 1 P 2 Maximum Needs 10 4 9 Current Needs 5 2 2 The system at t 0 is safe since the sequence

exists. 25

Example The system has three processes and 12 tape drives. t=t 0 P 1

Example The system has three processes and 12 tape drives. t=t 0 P 1 P 2 Maximum Needs 10 4 9 Current Needs 5 2 2 P 2 requests one more drive t=t 1 P 0 P 1 P 2 Maximum Needs 10 4 9 Current Needs 5 2 3 The system at t 1 is no longer safe since • P 1 requests 2 more tape drives, finishes and releases 4 drives. • However 4 drives are not sufficient for P 0 or P 2 complete its operation and would result in a deadlock. 26

CENG 334 Introduction to Operating Systems Real-world cases Topics: • Race conditions • Priority

CENG 334 Introduction to Operating Systems Real-world cases Topics: • Race conditions • Priority Inversion Erol Sahin Dept of Computer Eng. Middle East Technical University Ankara, TURKEY URL: http: //kovan. ceng. metu. edu. tr/ceng 334 27

Therac-25 Computer-controlled radiation therapy machine In operation between 1983 and 1987, 11 installations Adapted

Therac-25 Computer-controlled radiation therapy machine In operation between 1983 and 1987, 11 installations Adapted from Matt Welsh’s (Harvard University) slides. 28

Therac-25 Capable of delivering electron and photon (X-Ray) treatments Completely computer controlled No hardware

Therac-25 Capable of delivering electron and photon (X-Ray) treatments Completely computer controlled No hardware interlocks to prevent misconfigurations or overdoses! All software written in PDP-11 assembly language Cryptic error messages delivered to operator console “Malfunction 23” No documentation of these error codes No indication of which errors are potentially life-threatening Lots of smoke and mirrors by the manufacturer Claimed that 10 -11 chance of delivering wrong dose to patient No justification for this claim in the safety analysis documents Adapted from Matt Welsh’s (Harvard University) slides. 29

Accidents On several occasions between June '85 and Jan '87 Massive overdoses to six

Accidents On several occasions between June '85 and Jan '87 Massive overdoses to six people Some of these were lethal Typical theraputic doses in the 200 rad range Several overdoses delivered energy of 15, 000 – 20, 000 rads Various lawsuits, all settled out of court Initially, manufacturer claimed that overdoses were impossible Adapted from Matt Welsh’s (Harvard University) slides. 30

The problem Therac-25 operator console layout. The lethal computer error occurs when the operator

The problem Therac-25 operator console layout. The lethal computer error occurs when the operator accidentally sets the field (here in red) to "X", notices her mistake, then changes it to "E". Adapted from Matt Welsh’s (Harvard University) slides. 31

Race Condition #1 After some trial and error, it was discovered that overdose could

Race Condition #1 After some trial and error, it was discovered that overdose could be caused by operator editing the dosage on the console too quickly Operator would enter dosage on console Move cursor to bottom of screen, then move cursor back up to edit dosage “Treat” task Periodically checks “entry done” flag If flag is set, call subroutine to configure the magnets Configuring magnets takes about 8 sec “Magnet” task Called periodically to check if magnets are ready Checks if edits have been made to dosage If so, exits back to calling subroutine to restart the process Critical bug: Only checks if edits made on the first call! How this led to overdose: Operator enters dosage: Triggers magnet setting routine Operator edits dosage while the magnets are being configured Magnet routine does notice edits have been made after first call Adapted from Matt Welsh’s (Harvard University) slides. 32

Race Condition #2 Second bug – totally different causes from the first THERAC-25 has

Race Condition #2 Second bug – totally different causes from the first THERAC-25 has a “turntable” aperature that moves certain elements into the path of the beam Beam Computer controls position of turntable X-Ray field flattner Electron scan magnet Field light position (no electron beam) Field light mode used to position beam on patient No electron beam expected, instead, a light simulates the beam position Problem: Unfiltered beam exposed to patients on several occasions! Adapted from Matt Welsh’s (Harvard University) slides. 33

Race Condition #2 1) Prescription entered on console 2) Operator must press “set” button

Race Condition #2 1) Prescription entered on console 2) Operator must press “set” button to configure turntable 3) “Set up test” task runs periodically to check position of turntable Increments a variable “Class 3” on each iteration If “Class 3 == 0”, everything is ready and the dosage can begin Otherwise, a series of interlock checks are performed to ensure turntable in the correct position These checks will set Class 3 to 0 when they are complete Can you spot the bug? Adapted from Matt Welsh’s (Harvard University) slides. 34

Race Condition #2 The bug: “Class 3” variable is 8 bits wide After 256

Race Condition #2 The bug: “Class 3” variable is 8 bits wide After 256 iterations of “set up test” routine, overflows and becomes zero! So, interlocking checks will not be performed Operator must press “set” button during the short interval that Class 3 overflows Fix: Set “Class 3” to some nonzero value, rather than incrementing it Why was this done? Probably because “inc” instruction was easy enough. . . http: //sunnyday. mit. edu/papers/therac. pdf Adapted from Matt Welsh’s (Harvard University) slides. 35

Mars Pathfinder July 4, 1997 landing on Martian surface, followed by expeditions by Sojourner

Mars Pathfinder July 4, 1997 landing on Martian surface, followed by expeditions by Sojourner rover Series of software glitches started a few days after landing Eventually debugged and patched remotely from Earth! Read the full story at: http: //www. ddj. com/184411097 Adapted from Matt Welsh’s (Harvard University) slides. 36

Vx. Works Operating System Developed by Wind River Systems – premier real time OS

Vx. Works Operating System Developed by Wind River Systems – premier real time OS Multiple tasks, each with an associated priority Higher priority tasks get to run before lower-priority tasks Information bus – shared memory area used by various tasks Thread must obtain mutex to write data to the info bus – a monitor Weather Data Thread Communication Thread Obtain mutex; write data Information Bus Thread Wait for mutex to read data Mutex Information Bus Adapted from Matt Welsh’s (Harvard University) slides. 37

Vx. Works Operating System Developed by Wind River Systems – premier real time OS

Vx. Works Operating System Developed by Wind River Systems – premier real time OS Multiple tasks, each with an associated priority Higher priority tasks get to run before lower-priority tasks Information bus – shared memory area used by various tasks Thread must obtain mutex to write data to the info bus – a monitor Weather Data Thread Communication Thread Information Bus Thread Free mutex Mutex Information Bus Adapted from Matt Welsh’s (Harvard University) slides. 38

Vx. Works Operating System Developed by Wind River Systems – premier real time OS

Vx. Works Operating System Developed by Wind River Systems – premier real time OS Multiple tasks, each with an associated priority Higher priority tasks get to run before lower-priority tasks Information bus – shared memory area used by various tasks Thread must obtain mutex to write data to the info bus – a monitor Weather Data Thread Communication Thread Information Bus Thread Lock mutex and read data Mutex Information Bus Adapted from Matt Welsh’s (Harvard University) slides. 39

Priority Inversion What happens when threads have different priorities? Low priority Weather Data Thread

Priority Inversion What happens when threads have different priorities? Low priority Weather Data Thread Med Priority Communication Thread High priority Information Bus Thread Mutex Information Bus Adapted from Matt Welsh’s (Harvard University) slides. 40

Priority Inversion What happens when threads have different priorities? Interrupt! Schedule comm thread. .

Priority Inversion What happens when threads have different priorities? Interrupt! Schedule comm thread. . . long running operation Low priority Weather Data Thread Med Priority Communication Thread High priority Information Bus Thread Mutex Information Bus Adapted from Matt Welsh’s (Harvard University) slides. 41

Priority Inversion What happens when threads have different priorities? Comm thread runs for a

Priority Inversion What happens when threads have different priorities? Comm thread runs for a long time Comm thread has higher priority than weather data thread But. . . the high priority info bus thread is stuck waiting! This is called priority inversion Low priority Weather Data Thread Med Priority Communication Thread High priority Information Bus Thread Mutex Information Bus Adapted from Matt Welsh’s (Harvard University) slides. 42

What is the fix? Problem with priority inversion: A high priority thread is stuck

What is the fix? Problem with priority inversion: A high priority thread is stuck waiting for a low priority thread to finish its work In this case, the (medium priority) thread was holding up the low-prio thread General solution: Priority inheritance If waiting for a low priority thread, allow that thread to inherit the higher priority High priority thread “donates” its priority to the low priority thread Why does this fix the problem? Medium priority comm task cannot preempt weather task Weather task inherits high priority while it is being waited on Adapted from Matt Welsh’s (Harvard University) slides. 43

How was this problem fixed? JPL had a replica of the Pathfinder system on

How was this problem fixed? JPL had a replica of the Pathfinder system on the ground Special tracing mode maintrains logs of all interesting system events e. g. , context switches, mutex lock/unlock, interrupts After much testing were able to replicate the problem in the lab Vx. Works mutex objects have an optional priority inheritance flag Engineers were able to upload a patch to set this flag on the info bus mutex After the fix, no more system resets occurred Lessons: Automatically reset system to “known good” state if things run amuck Far better than hanging or crashing Ability to trace execution of complex multithreaded code is useful Think through all possible thread interactions carefully!! Adapted from Matt Welsh’s (Harvard University) slides. 44