The Sombrero Single Address Space Operating System Prototype
The Sombrero Single Address Space Operating System Prototype A Testbed for Evaluating Distributed Persistent System Concepts and Implementation Alan Skousen Donald Miller Computer Science and Engineering Department Arizona State University E-mail: {alan. skousen, donald. miller}@asu. edu http: //www. eas. asu. edu/~sasos 9/29/2020 ASU 64 -bit OS Group 1
OUTLINE • Single Address Spaces Single Address Space Operating Systems and Sombrero – who are these guys? • Distributed Persistent System Issues in Sombrero Design and Implementation • • • Software Engineering Benefits Consistency Management Persistence Fault Tolerance Programming Language-Related Issues • Conclusions 9/29/2020 ASU 64 -bit OS Group 2
Single Address Space Characteristics • Virtual Addresses can be permanently and uniquely bound to all code and data objects – VAs can serve as unique names – VA space can serve as the only namespace • The Virtual Address namespace spans all levels of the storage hierarchy on every node – All Physical storage can be viewed as a hierarchy of caches for the contents of virtual addresses • The Virtual Address namespace is manipulated directly by the CPU and access to it is controlled directly by memory and protection management hardware – the CPU can directly enforce principal protection and resource allocation access policies on all objects defined in the system as it manipulates virtual addresses – every VA is tied to a specific access policy 9/29/2020 ASU 64 -bit OS Group 3
Useful Features Provided by a SASOS • • Address translations remain the same for all programs Threads are free to travel throughout the VA space with no changes in the environment in which they are running in except for protection context Network-wide communication requires no prior/additional setup Internal pointers and pointers into other objects remain the same across all levels of storage and all programs – marshalling, flattening and dynamic linking not needed • Persistence without use of a separate file system – Where data is accessed, organized and addressed differently • Protection by restricting what a computation is allowed to access rather than what it is allowed to address – managing IPC is reduced to managing protection 9/29/2020 ASU 64 -bit OS Group 4
Single Address Space Operating Systems (SASOSs) • Software SASOSs – Use stock 64 -bit RISC processors – Require additional software communication protocols for protection – Examples: Opal and Mungi • Hardware supported SASOSs – Require additional CPU hardware for protection – Examples: AS/400, Monads, Sombrero A HW-supported SASOS can take fuller advantage of single address space characteristics. 9/29/2020 ASU 64 -bit OS Group 5
9/29/2020 ASU 64 -bit OS Group 6
Sombrero • Designed to take advantage of properties of a single address space • Major Abstractions – Memory Objects, Threads, Protection Domains and Principals • Has a hardware and a software design – Range Protection Lookaside Buffer (RPLB) – Flat, modular peer-level architecture 9/29/2020 7
RPLB HW FUNCTIONALITY • • • Separation of Address Translation and Address Protection Functions Hardware Caching of Allowed Protection Domain Crossings Protection Domains for Threads Distinct from the General Protection Context Implicit Domain Crossing using Ordinary Instructions Protection for Variable Granularity Object Sizes RPLB Stores Triples of the form: <protection domain tag, resource accessed, access rights> 9/29/2020 ASU 64 -bit OS Group 8
Sombrero Current Architecture Single Node • Kernel Services distributed among executive protection domains • No central kernel and no hardware protected kernel mode • A few Protection Domain Lock Registers name the protection domain that can access sensitive protected instructions and registers 9/29/2020 ASU 64 -bit OS Group 9
Sombrero Prototype • • • “Nearly completed” on 2 sets of Alpha boxes Contains a subset of full design Semantics of system interface are as in the final version Prototype emulates an RPLB – uses TLB to provide initial protection faults Uses Windows NT for windowing and file system Implementation overview – Reachable address space extended – specialized NT process – PALCode recognizes a Sombrero process and forwards TLB misses outside the normal range to a Sombrero extension of PALCode – Pager in NT user space forms correct TLB entry and handles miss – RPLB emulated by PALCode extension – Individual threads assigned to separate PDs within the specialized process – Provides a functional protected single address space similar that which Sombrero would provide. 9/29/2020 ASU 64 -bit OS Group 10
Distributed Persistent System Issues in Sombrero Design and Implementation • Software Engineering Benefits • Consistency Management • Persistence • Fault Tolerance • Programming Language - related Issues – Types/Scope/Protection Domains/Linking 9/29/2020 ASU 64 -bit OS Group 11
Distributed Persistent System Issues Software Engineering Benefits • Reducing the Cost of Software Development by Reducing Software Complexity – – IDC versus IPC Communication via Shared Memory vs. Message Passing System Services via Procedure Call vs. System Call Local Objects and Remote Objects in common VA space • Performance should be slightly better – not the major goal 9/29/2020 ASU 64 -bit OS Group 12
Software Engineering Benefits Software Complexity • Complexity Metrics – Mc. Cabe’s Cyclomatic Complexity – Halstead’s Software Science – Lines of Code • Test Case – an Object-Oriented Database • Management System Other Design, Testing and Maintenance Metrics 9/29/2020 ASU 64 -bit OS Group 13
//Header file used by the client and server: /* Definitions needed by clients and servers. */ #define MAX_PATH 255/* maximum length of a file name */ #define BUF_SIZE 1024/* how much data to transfer at once */ #define FILE_SERVER 243/* file server's network address */ /* Definitions of the allowed operations. */ #define CREATE 1/* create a new file */ #define READ 2/* read a piece of a file and return it */ #define WRITE 3/* write a piece of a file */ #define DELETE 4/* delete an existing file */ /* Error codes. */ #define OK 0/* operation performed correctly */ #define E_BAD_OPCODE -1/* unknown operation requested */ #define E_BAD_PARAM -2/* error in a parameter */ #define E_10 -3/* disk error or other I/O error */ /* Definition of the message format. */ struct message { long source; /* sender's identity */ long dest; /* receiver's identity */ long opcode; /* which operation: CREATE, READ, etc. */ long count; /* how many bytes to transfer */ long offset; /* where in file to start reading or writing */ long extra 1; /* extra field */ long extra 2; /* extra field */ long result; /* result of the operation */ char name[MAX_PATH]; /* name of the file being operated on */ char data[BUF_SIZE]; /* data to be read or written */ }; // Process oriented file SERVER: #include <header. h> void main(void) { struct message m 1, m 2; /* incoming and outgoing messages */ int r; /* result code */ initialize(); while (1) { receive (FILE_SERVER, &m 1); /* server runs forever */ switch(m 1. opcode) { case CREATE: r = do_create(&m 1, &m 2); break; case READ: r = do_read(&m 1, &m 2); break; case WRITE: r = do_write(&m 1, &m 2); break; case DELETE: r = do_delete(&m 1, &m 2); break; default: r = E_BAD_OPCODE; } m 2. result = r; /* return result to client */ send(m 1. source, &m 2); /* send reply */ } } 9/29/2020 //Process oriented CLIENT uses file server to copy a file: #include <header. h> int copy(char *src, char *dst) /* procedure to copy file using the server */ { struct message m 1; /* message buffer */ long position; /* current file position */ long client = 110; /* client's address */ /* prepare for execution */ position = 0; do { /* Get block of data from source file */ m 1. opcode = READ; /* operation is a read */ m 1. offset = position; /* current position in the file */ m 1. count = BUF_SIZE; /* how many bytes to read */ strcpy(&m 1. name, src); /* copy name of file to be read to message */ send(FILE_SERVER, &m 1); /* send message to the file server */ receive(client, &m 1); /* block waiting for the reply */ Software Engineering Benefits Process-Oriented OS File Copy vs. SASOS Memory Object Copy /* Write the data just received to the destination file */ m 1. opcode = WRITE; /* operation is a write */ m 1. offset = position; /* current position in the file */ m 1. count = m 1. result; /* how many bytes to write */ strcpy(&m 1. name, dst); /* copy name of file to be written to buf */ send(FILE_SERVER, &m 1); /* send the message to the file server */ receive(client, &m 1); /* block waiting for the reply */ position += m 1. result; /* m 1. result is the number of bytes written */ } while (m 1. result > 0); /* iterate until done */ return(m 1. result >= 0 ? OK : m 1. result); /* return OK or error code */ } //SASOS Memory object copy int copy(char *src, char *dst) { FILE_OBJECT *from, *to; //address() is a function that obtains an object's address from the nameservice from = address(src); // allocate a new memory object to = new FILE_OBJECT(dst, PERSIST, from->Size()); memcpy(to, from->Size()); } ASU 64 -bit OS Group 14
Distributed Persistent System Issues Consistency Management in Sombrero • • A consistency model is essentially a contract between an application and memory – the memory will work correctly if the software follows certain rules in its assumptions. – Example: Sequential consistency (simplified): applications all see the writes by different nodes in the same order and see the writes on a given node in the order they occur on that node. Sombrero’s Implementation of Consistency Management – – Objective Some General Features and Concepts Consistency models supported Overall Structure: Interaction between protection, consistency, domain and communication management services – Data Structures for distributed copy set management – Kernel Protection Domain Control Structures – Other Issues: Node Awareness/Consistency Service Idempotence/Trust and Proxy Threads/Status of this work 9/29/2020 ASU 64 -bit OS Group 15
Consistency Management in Sombrero Objective • Manage Consistency in a Distributed VA Space – Resource Access – Control Transfer – Kernel Function distribution • Powerful, Extensible and Scalable 9/29/2020 ASU 64 -bit OS Group 16
Consistency Management in Sombrero Some General Features and Concepts • • The common VA space distributed over all nodes and all levels of the memory cache hierarchy serves as a substrate consistency management can be transparent, built in, automatic and provide no extra burden on a developer no middleware and no extra DSM space VA spaces don’t define protection and pages don’t determine object granularity consistency is with respect to objects and data transport uses pages – management and granularity are different Consistency is implemented by setting and unsetting tokens that enable/disable {rwx} privileges for a memory object and determine the existence of a protection domain on a given node Consistency mechanism manages consistency of replicates across the net – does not address local r/w synchronization 9/29/2020 ASU 64 -bit OS Group 17
Consistency Management in Sombrero Consistency Models Supported • • Sequential – the default MRMW (Multiple Reader Multiple Writer) – r/w tokens enabled wherever data is • • Weak Implementation Features – Developers can extend base classes to add their own consistency services, e. g. , causal from sequential or CREW from weak – Application programmers can build transactional semantics by using mutexes – One consistency service per memory object and a consistency service can support many memory objects 9/29/2020 ASU 64 -bit OS Group 18
Consistency Management in Sombrero Overall Structure • • Client on Node X tries to access a memory object and there’s an RPLB miss Protection Service (PS) searches protection caches and if necessary contacts the Domain Manager for user access policy If access is allowed check for enabling token If token is missing contact Node X’s Consistency Service (CS) – address of appropriate CS service found in MOCB – call mechanism depends on whether CS is system or user defined If required, Node X’s CS contacts Node Y’s CS, etc. , via Communications Service. At each node CS checks with PS When satisfied with state of object CS calls back PS on Node X If CS has to move pages around it contacts Paging Service who packages them up and sends them to Communication Service 9/29/2020 ASU 64 -bit OS Group 19
Consistency Management in Sombrero Distributed Object Copy Set Management Last Known Writer Graph Pruning of Modified Page Cache Graph Pointers stored at each node keep track of where pages have been sent to/received from – these form edges of graphs 9/29/2020 Copy. Set Graph ASU 64 -bit OS Group 20
Consistency Management in Sombrero Kernel Protection Domain Control Structures • • • Kernel Protection Domain Control Structures represented by Surrogates on remote nodes – Same name (VA on inception node) – For efficiency – different VA for surrogate control block on remote nodes Simple Semantics – tokens used to either enable or disable GPDs and CPDs on a node Simple thread migration is a ‘for free’ side effect because in a single address space threads are consistent i. e. , have the same context wherever they execute 9/29/2020 ASU 64 -bit OS Group 21
Consistency Management in Sombrero Other Consistency Management Issues • Node Awareness – – • Consistency Services at a node must have some node awareness Applications are not required to be node aware Consistency is transparent (and ubiquitous) Idempotence, Trust and Proxy Threads – System-provided consistency managers are ‘idempotent’ and ‘trusted’ Idempotent calls will work and proxy threads not needed to protect caller’s data – User-provided consistency managers are ‘not trusted’ Idempotent calls may not be supported and proxy threads are needed to protect caller’s data – Proxy threads may still be needed sometimes to prevent thrashing storms caused by recursive thread migration during consistency management • Status of this work: logical design completed/coding in process 9/29/2020 ASU 64 -bit OS Group 22
Distributed Persistent System Issues Persistence • • Oversimplifying - Persistent Data is data that sticks around after the program that created it has exited and (typically) after the OS environment it was created in has shut down. A single virtual address space is by its nature a single persistent namespace - data bound to a VA is bound to it over all levels of the memory cache and for all time. The VA space is an abstraction that always exists at some level(s) of cache. A distributed single address space is automatically and ubiquitously a persistent store. • • If we flush the memory cache back to a persistent store - it’s contents are automatically persistent. In addition - no flattening or serialization is needed. The store is not RAM-Centric. Thread migration is a persistent mechanism in a single address space because there is a persistent copy of the thread’s state. 9/29/2020 ASU 64 -bit OS Group 23
Distributed Persistent System Issues Fault Tolerance • • • Sombrero defines an archival system-wide fault tolerant layer at the bottom of the memory hierarchy. If system is small enough system-wide checkpoints can be used. Else - consistent subset checkpoints are needed – these require data serialization where the issues are the same as for process-oriented systems. Fault tolerance in a single address space is no harder than (and no easier than) in process-oriented systems Since kernel structures can be replicated RAIN (Redundant Array of Inexpensive Nodes) is possible. 9/29/2020 ASU 64 -bit OS Group 24
Distributed Persistent System Issues Programming Language-Related Issues • • Types – Persistence is not a language issue (it’s an artifact of legacy process-oriented systems) Scope – Process scope has been replaced by a combination of global and protection domain scope. – Thread Scope associated with a Carrier Protection Domain is provided by the system. Threads really do have private data on their stacks. Protection Domains – Every Sombrero program is an instantiation of an Object Class and is implemented as a General Protection Domain. Linking is different in a single address space 9/29/2020 ASU 64 -bit OS Group 25
Conclusions • • A SASOS substrate can directly provide services that must be provided by Middleware in conventional client/server process-oriented systems. – Middleware is replaced by bottomware (underware) These services include Resource Access, Control Transfer and Communication. This vastly reduces the complexity of application software and the costs of its design. The combination of CPU-resident protection hardware and a SASOS that runs on it makes fuller use of the properties of a very large distributed address space than contemporary process-oriented systems. – Reduced cost for software development – Higher performance – Better support for distributed applications – A better match to the needs of real-time systems 9/29/2020 ASU 64 -bit OS Group 26
- Slides: 26