Advanced Operating Systems Lecture notes http gost isi

  • Slides: 42
Download presentation
Advanced Operating Systems Lecture notes http: //gost. isi. edu/555 Dr. Clifford Neuman University of

Advanced Operating Systems Lecture notes http: //gost. isi. edu/555 Dr. Clifford Neuman University of Southern California Information Sciences Institute Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Announcements v. Mid-term Grading Complete q TA will provide details about pick-up v. Assignment

Announcements v. Mid-term Grading Complete q TA will provide details about pick-up v. Assignment 4 q Due November 14 Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

CSci 555: Advanced Operating Systems Lecture 12 – November 09 2007 Kernels Continued, Scheduling,

CSci 555: Advanced Operating Systems Lecture 12 – November 09 2007 Kernels Continued, Scheduling, Fault Tolerance Real Time, Database Support Dr. Clifford Neuman University of Southern California Information Sciences Institute Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

The X-Kernel v Uof. Arizona, 1990. v Like V, communication services are critical. v

The X-Kernel v Uof. Arizona, 1990. v Like V, communication services are critical. v Machines communicating through internet. Heterogeneity! q The more protocols on user’s machine, the more resources are accessible. q v The x-kernel philosophy: provide infrastructure to facilitate protocol implementation. Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Virtual Protocols v. The x-kernel provide library of protocols. q Combined differently to access

Virtual Protocols v. The x-kernel provide library of protocols. q Combined differently to access different resources. q Example: § If communication between processes on the same machine, no need for any networking code. § If on the same LAN, IP layer skipped. Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

The X-Kernel : Process and Memory v ability to pass control and data efficiently

The X-Kernel : Process and Memory v ability to pass control and data efficiently between the kernel and user programs user data is accessible because kernel process executes in same address space vkernel process -> user process q q q sets up user stack pushes arguments user-stack access only user data v kernel -> user (245 usec), user -> kernel 20 usec on SUN 3/75 Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Communication Manager v Object-oriented infrastructure for implementing and composing protocols. v Common protocol interface.

Communication Manager v Object-oriented infrastructure for implementing and composing protocols. v Common protocol interface. v 2 abstract communication objects: q Protocols and sessions. q Example: TCP protocol object. § TCP open operation: creates a TCP session. § TCP protocol object: switches each incoming message to one of the TCP session objects. § Operations: demux, push, pop. Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

X-kernel Configuration UDP TCP RPC TCP UDP IP IP ETH Message Object Session Object

X-kernel Configuration UDP TCP RPC TCP UDP IP IP ETH Message Object Session Object Protocol Object Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE RPC

Message Manager v Defines single abstract data type: message. q Manipulation of headers, data,

Message Manager v Defines single abstract data type: message. q Manipulation of headers, data, and trailers that compose network transmission units. q Well-defined set of operations: § Add headers and trailers, strip headers and trailers, fragment/reassemble. q Efficient implementation using directed acyclic graphs of buffers to represent messages + stack data structure to avoid data copying. Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Mach v CMU (mid 80’s). v Mach is a microkernel, not a complete OS.

Mach v CMU (mid 80’s). v Mach is a microkernel, not a complete OS. v Design goals: As little as possible in the kernel. q Portability: most kernl code is machine independent. q Extensibility: new features can be implemented/tested alongside existing versions. q Security: minimal kernel specified and implemented in more secure way. q Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Mach Features v. OSs as Mach applications. v. Mach functionality: q Task and thread

Mach Features v. OSs as Mach applications. v. Mach functionality: q Task and thread management. q IPC. q Memory management. q Device management. Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Mach IPC v Threads communicate using ports. v Resources are identified with ports. v

Mach IPC v Threads communicate using ports. v Resources are identified with ports. v To access resource, message is sent to corresponding port. q Ports not directly accessible to programmer. q Need handles to “port rights”, or capabilities (right to send/receive message to/from ports). v Servers: manage several resources, or ports. Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Mach: ports vprocess port is used to communicate with the kernel. vbootstrap port is

Mach: ports vprocess port is used to communicate with the kernel. vbootstrap port is used for initialization when a process starts up. vexception port is used to report exceptions caused by the process. vregistered ports used to provide a way for the process to communicate with standard system servers. Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Protection v. Protecting resources against illegal access: q Protecting port against illegal sends. v.

Protection v. Protecting resources against illegal access: q Protecting port against illegal sends. v. Protection through capabilities. q Kernel controls port capability acquisition. q Different from Amoeba. Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Capabilities 1 v Capability to a port has field specifying port access rights for

Capabilities 1 v Capability to a port has field specifying port access rights for the task that holds the capability. q Send rights: threads belonging to task possessing capability can send message to port. q Send-once rights: allows at most 1 message to be sent; after that, right is revoked by kernel. q Receive rights: allows task to receive message from port’s queue. § At most 1 task, may have receive rights at any time. § More than 1 task may have sned/send-once rights. Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Capabilities 2 v. At task creation: q Task given bootstrap port right: send right

Capabilities 2 v. At task creation: q Task given bootstrap port right: send right to obtain services of other tasks. q Task threads acquire further port rights either by creating ports or receiving port rights. Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Port Name Space Task T (user level) System call referring to right on port

Port Name Space Task T (user level) System call referring to right on port i Kernel i Port i’s rights. . Mach’s port rights stored inside kernel. . Tasks refer to port rights using local id’s valid in the task’s local port name space. . Problem: kernel gets involved whenever ports are referenced. Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Communication Model v. Message passing. v. Messages: fixed-size headers + variable-length list of data

Communication Model v. Message passing. v. Messages: fixed-size headers + variable-length list of data items. Header Pointer to out-of Port rights T In-line data T T line data Header: destination port, reply port, type of operation. T: type of information. Port rights: send rights: receiver acquires send rights to port. Receive rights: automatically revoked in sending task. Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Ports v. Mach port has message queue. q Task with receive rights can set

Ports v. Mach port has message queue. q Task with receive rights can set port’s queue size dynamically: flow control. q If port’s queue is full, sending thread is blocked; send-once sender never blocks. v. System calls: q Send message to kernel port. q Assigned at task creation time. Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Task and Thread Management v Task: execution environment (address space). v Threads within task

Task and Thread Management v Task: execution environment (address space). v Threads within task perform action. v Task resources: address space, threads, port rights. v PAPER: q How Mach microkernel can be used to implement other OSs. q Performace numbers comparing 4. 3 BSD on top of Mach and Unix kernels. Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

CSci 555: Advanced Operating Systems Lecture 12 – November 09 2007 Scheduling, Fault Tolerance

CSci 555: Advanced Operating Systems Lecture 12 – November 09 2007 Scheduling, Fault Tolerance Real Time, Database Support Dr. Clifford Neuman University of Southern California Information Sciences Institute Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Scheduling and Real-Time systems v. Scheduling q Allocation of resources at a particular point

Scheduling and Real-Time systems v. Scheduling q Allocation of resources at a particular point in time to jobs needing those resources, usually according to a defined policy. v. Focus q We will focus primarily on the scheduling of processing resources, though similar concepts apply the scheduling of other resources including network bandwidth, memory, and special devices. Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Parallel Computing - General Issues v Speedup - the final measure of success q

Parallel Computing - General Issues v Speedup - the final measure of success q Parallelism vs Concurrency § Actual vs possible by application q Granularity § Size of the concurrent tasks § Reconfigurability q Number of processors q Communication cost q Preemption v. non-preemption q Co-scheduling § Some things better scheduled together Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Shared Memory Multi-Processing v Includes use of distributed shared memory, and shared memory multi-processors

Shared Memory Multi-Processing v Includes use of distributed shared memory, and shared memory multi-processors v Processors usually tightly coupled to memory, often on a shared bus. Programs communicated through shared memory locations. v For SMPs cache consistency is the important issue. In DSM it is memory coherence. q One level higher in the storage hierarchy v Examples § Sequent, Encore Multimax, DEC Firefly, Stanford DASH Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Where is the best place for scheduling v Application is in best position to

Where is the best place for scheduling v Application is in best position to know its own specific scheduling requirements q Which threads run best simultaneously q Which are on Critical path q But Kernel must make sure all play fairly v MACH Scheduling q Lets process provide hints to discourage running q Possible to hand off processor to another thread § Makes easier for Kernel to select next thread § Allow interleaving of concurrent threads q Leaves low level scheduling in Kernel q Based on higher level info from application space Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Scheduler activations v. User level scheduling of threads q Application maintains scheduling queue v.

Scheduler activations v. User level scheduling of threads q Application maintains scheduling queue v. Kernel allocates threads to tasks q Makes upcall to scheduling code in application when thread is blocked for I/O or preempted q Only user level involved if blocked for critical section v. User level will block on kernel calls q Kernel returns control to application scheduler Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Distributed-Memory Multi-Processing v Processors coupled to only part of the memory q Direct access

Distributed-Memory Multi-Processing v Processors coupled to only part of the memory q Direct access only to their own memory v Processors interconnected in mesh or network q Multiple hops may be necessary v May support multiple threads per task v Typical characteristics q Higher communication costs q Large number of processors q Coarser granularity of tasks v Message passing for communication Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Condor v. Identifies idle workstations and schedules background jobs on them v. Guarantees job

Condor v. Identifies idle workstations and schedules background jobs on them v. Guarantees job will eventually complete Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Condor v. Analysis of workstation usage patterns q Only 30% v. Remote capacity allocation

Condor v. Analysis of workstation usage patterns q Only 30% v. Remote capacity allocation algorithms q Up-Down algorithm § Allow fair access to remote capacity v. Remote execution facilities q Remote Unix (RU) Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Condor v. Leverage: performance measure q Ratio of the capacity consumed by a job

Condor v. Leverage: performance measure q Ratio of the capacity consumed by a job remotely to the capacity consumed on the home station to support remote execution v. Checkpointing: save the state of a job so that its execution can be resumed Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Condor - Issues v. Transparent placement of background jobs v. Automatically restart if a

Condor - Issues v. Transparent placement of background jobs v. Automatically restart if a background job fails v. Users expect to receive fair access v. Small overhead Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Condor - scheduling v. Hybrid of centralized static and distributed approach v. Each workstation

Condor - scheduling v. Hybrid of centralized static and distributed approach v. Each workstation keeps own state information and schedule v. Central coordinator assigns capacity to workstations q Workstations use capacity to schedule Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Prospero Resource Manager - 3 entities v One or more system managers q Each

Prospero Resource Manager - 3 entities v One or more system managers q Each manages subset of resources q Allocates resources to jobs as needed v A job manager associated with each job q Identifies resource requirements of the job q Acquires resources from one or more system managers q Allocates resources to the job’s tasks v A Node manager on each node q Mediates access to the nodes resources Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

The Prospero Resource Manager Read stdin, Write stdout, stderr User’s workstation % appl Filesystem

The Prospero Resource Manager Read stdin, Write stdout, stderr User’s workstation % appl Filesystem file 1 file 2 • • • Filesystem Terminal I/O file 1 file 2 • • • T 3 Node T 1 Read file Write file A) User invokes an application program on his workstation. T 2 Node b) The program begins executing on a set of nodes. Tasks perform terminal and file I/O on the user’s workstation. Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Advantages of the PRM v. Scalability q q System manager does not require detailed

Advantages of the PRM v. Scalability q q System manager does not require detailed job information Multiple system managers v. Job manager selected for application q q Knows more about job’s needs than the system manager Alternate job managers useful for debugging, performance tuning v. Abstraction q q Job manager provides a single resource allocator for the job’s tasks Single system model Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Real time Systems v. Issues are scheduling and interrupts q Must complete task by

Real time Systems v. Issues are scheduling and interrupts q Must complete task by a particular deadline q Examples: § Accepting input from real time sensors § Process control applications § Responding to environmental events v. How does one support real time systems q If short deadline, often use a dedicated system q Give real time tasks absolute priority q Do not support virtual memory § Use early binding Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Real time Scheduling v To initiate, must specify q Deadline q Estimate/upper-bound on resources

Real time Scheduling v To initiate, must specify q Deadline q Estimate/upper-bound on resources v System accepts or rejects q If accepted, agrees that it can meet the deadline q Places job in calendar, blocking out the resources it will need and planning when the resources will be allocated v Some systems support priorities q But this can violate the RT assumption for already accepted jobs Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Fault-Tolerant systems v. Failure probabilities q Hierarchical, based on lower level probabilities q Failure

Fault-Tolerant systems v. Failure probabilities q Hierarchical, based on lower level probabilities q Failure Trees q Add probabilities where any failure affects you – Really (1 - ((1 - lambda)(1 -lambda) (1 - lambda))) q Multiply probabilities if all must break § Since numbers are small, this reduces failure rate q Both failure and repair rate are important Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Making systems fault tolerant v. Involves masking failure at higher layers q Redundancy q

Making systems fault tolerant v. Involves masking failure at higher layers q Redundancy q Error correcting codes q Error detection v. Techniques q In hardware q Groups of servers or processors execute in parallel and provide hot backups v. Space Shuttle Computer Systems exampls v. RAID example Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Types of failures v. Fail stop q Signals exception, or detectably does not work

Types of failures v. Fail stop q Signals exception, or detectably does not work v. Returns wrong results q Must decide which component failed v. Byzantine q Reports difficult results to different participants q Intentional attacks may take this form Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

Recovery v. Repair of modules must be considered q Repair time estimates v. Reconfiguration

Recovery v. Repair of modules must be considered q Repair time estimates v. Reconfiguration q Allows one to run with diminished capacity q Improves fault tolerance (from catastrophic failure) Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE

OS Support for Databases v. Example of OS used for particular applications v. End-to-end

OS Support for Databases v. Example of OS used for particular applications v. End-to-end argument for applications q Much of the common services in OS’s are optimized for general applications. q For DBMS applications, the DBMS might be in a better position to provide the services § Caching, Consistency, failure protection Copyright © 1995 -2005 Clifford Neuman and Dongho Kim - UNIVERSITY OF SOUTHERN CALIFORNIA - INFORMATION SCIENCES INSTITUTE