Single Address Spaces Single Address Space Operating Systems

Single Address Spaces Single Address Space Operating Systems and Sombrero Donald S. Miller Computer Science and Engineering Department Arizona State University 9/25/2020 ASU 64 -bit OS Group 1

OUTLINE • • What is a Single Address Space? What is a Single Address Space Operating System (SASOS)? Why build a SASOS now? Essence of the Single Address Space Paradigm SASOSs vs. Process-Oriented OSs - a more detailed look SASOSs with and without Additional Hardware Support Sombrero - a Hardware-Supported SASOS - Design Overview – Basic Abstractions/HW design/SW Protection Data Structures – OS Structure and System Architecture/Distributed System Design • • • SASOSs built on stock RISC Processors - Opal & Mungi SASOSs with additional protection HW - Monads & AS/400 SASOS Applications Current Trends that Favor a Hardware-Supported SASOS Summary 9/25/2020 ASU 64 -bit OS Group 2

Process-Oriented Operating Systems (Multiple Address Space Operating Systems) and Single Address Space Operating Systems (SASOSs) • Process-Oriented operating systems – Multiple virtual address namespaces – Unix® , NT™ , MVS • Single address space operating systems – Single VA namespace – Opal, Mungi, Grasshopper, Monads, AS/400, Sombrero • SASOSs feasible nowadays because of 64 -bit CPUs • Paradigm is different 9/25/2020 ASU 64 -bit OS Group 3

Why Single Address Space Operating Systems? • Recent 64 -bit Microprocessors have made an 18. 4 Quintillion Byte Address Space Potentially Available • Current Process-Oriented Operating Systems and Processor Page-based Protection HW are Designed to Conserve Virtual Addresses by Reusing them • These Facts have led to Investigations of Alternative Operating System and Processor Architecture Models • One such Model is to establish a Single Address Space in which all code and data reside and then Design an Operating System Specifically to run in this Environment • These Single Address Space OSs are called SASOSs 9/25/2020 ASU 64 -bit OS Group 4

Characteristics of a Single Address Space • Virtual Addresses can be permanently and uniquely bound to all code and data objects – VAs can serve as unique names – VA space can serve as the only namespace • The Virtual Address namespace spans all levels of the storage hierarchy on every node – All Physical storage can be viewed as a hierarchy of caches for the contents of virtual addresses • The Virtual Address namespace is manipulated directly by the CPU and access to it is controlled directly by memory and protection management hardware – the CPU can directly enforce access policy on all objects defined in the system as it manipulates virtual addresses 9/25/2020 ASU 64 -bit OS Group 5

Advantages that can be Provided by a SASOS • Address translations remain the same for all programs • Threads are free to travel throughout the VA space with no changes in the environment in which they are running in except for protection context • Network-wide communication requires no prior or additional setup • Internal pointers and pointers into other objects remain the same across all levels of storage and all programs – marshalling, flattening and dynamic linking not needed • Persistence without use of a separate file system • Protection by restricting what a computation is allowed to access rather than what it is allowed to address – managing IPC is reduced to managing protection 9/25/2020 ASU 64 -bit OS Group 6

Advantages that can be Provided by a SASOS (continued) • SASOSs increase the available choices – for structuring applications – for structuring the operating system – for sharing, protecting and storing data – for communication between programs • Fundamental Issue - how to structure an OS to provide – a simple program development environment – high performance in a system where conserving address space is no longer a driving concern 9/25/2020 ASU 64 -bit OS Group 7

Why build a SASOS now? • What’s changed? • What’s the reason for building a SASOS? • Why aren’t people building them? 9/25/2020 ASU 64 -bit OS Group 8

What’s Changed? • 64 -bit RISC processors are available. a paradigm shift to single address space operation is possible • Modern software has become increasingly large and complex increased SW engineering efforts are needed to reduce the costs of program development • Performance increases from reducing chip feature sizes will begin to decline in about 10 years Increased performance will have to come from architectural changes 9/25/2020 ASU 64 -bit OS Group 9

What’s the reason for building a SASOS? • Reducing the costs of program development – simpler mechanisms to support sharing and communication between programs • common naming mechanisms • system level support for OOD and OOP • Higher performance – gains from architectural changes • Increased support for distributed applications – Object-oriented DDBMS systems – CAD/CAM systems – Distributed parallel computations • Better match to needs of real-time applications 9/25/2020 ASU 64 -bit OS Group 10

Why aren’t people building them? • Going from multiple to single address spaces is a paradigm shift -there are costs associated with the following in a single address space – Understanding the paradigm shift – Learning programming differences – Mastering new approaches to HW/SW design • The CPU protection hardware necessary to make a SASOS viable introduces incompatible hardware – new software may not run on the old hardware consequent initial economic disadvantages. 9/25/2020 ASU 64 -bit OS Group 11

The Multiple to Single Address Space Paradigm Shift • PARADIGM Constellation of beliefs, values, techniques and so on shared by the members of a given community that implicitly define the legitimate problems and methods of a research field for succeeding generations of practitioners 1 • Shifting from multiple address spaces to a single address space constitutes a paradigm shift – brought about because in a single address space the set of virtual addresses can be used as a system-wide namespace So - What’s a namespace? __________ 1. The Structure of Scientific Revolutions, 3 rd Edition, Thomas S. Kuhn, University of Chicago Press, 1996. 9/25/2020 ASU 64 -bit OS Group 12

Names, Namespaces and Single Address Spaces • A name is a string that identifies an object. • A binding is a mapping between a name and an object. • A namespace is a set of bindings of names to objects. Alternatively, it’s a domain of possible names. • Namespaces include file names, IP numbers, port numbers and addresses in a virtual address space. A single address space is a namespace consisting of the entire range of virtual addresses. Each virtual address is unique and is permanently bound to a byte of a data object. 9/25/2020 ASU 64 -bit OS Group 13

Essence of the Multiple to Single Address Space Paradigm Shift are Going names object from which namespaces multiple in determined by the logical structures they are associated with, e. g. , process, file or record, or by the physical container that holds them, e. g. , cache, RAM or disk Going to asinglenetwork-widenamespaceinwhichobject names are determined by the unique virtual addresses used to access them Since VAs are directly operated upon by CPUs, the focus shifts from naming objects by the changing logical or physical structures associated with them to using VAs as the primary identifying/locating /accessing mechanism for all objects wherever they are and whenever they are used. 9/25/2020 ASU 64 -bit OS Group 14

Single Address Space vs. Multiple Address Space Paradigms Two sets of distinguishing characteristics • Features that differentiate between all SASOSs and process-oriented operating systems • Additional features that differentiate between HW-supported SASOSs and SASOSs built on stock RISC processors 9/25/2020 ASU 64 -bit OS Group 15

Single Address Space vs. Multiple Address Space Paradigms Process-Oriented OSs • Each process runs in it’s own VA Program VA space on a single node - VAs are Space context dependent Binding of • Process boundaries encapsulate and bind temporary VA spaces, Major Abstractions execution state, objects, protection domains and principals Persistent • Persistent storage is in a separate namespace - the file system Storage • Threads encapsulated within a process - domain crossing and Thread Mobility machine migration without an address space or thread context switch essentially impossible 9/25/2020 4: 22: 01 PM SASOSs • All programs run in the same VA namespace which spans all nodes VAs are not context dependent • Execution state, objects, protection domains and principals are not bound to VAs and hence not bound to each other • Persistent storage is also in the same virtual address namespace • Threads can travel throughout the virtual address space without an address space switch or a thread context switch ASU 64 -bit OS Group 16 16

Single Address Space vs. Multiple Address Space Paradigms (continued) Process-Oriented OSs • Message-based communication Data Exchange or RPC on top of message passing is frequently required and Synchronization for data exchange and synchronization • IPC using shared memory Shared requires prior and additional Memory setup and mapping Server Options Task and Data Migration 9/25/2020 SASOSs • Shared memory is the most prevalent mechanism for data exchange and synchronization • No prior or additional setup necessary for programs to communicate using shared memory • Servers must be active • Servers can be active or passive • Task and data migration is difficult - involves creating a new namespace for data (DSM) and dealing with distinct machine namespaces for code • Task and data migration is straight -forward - objects and threads exist at the same virtual addresses on all nodes ASU 64 -bit OS Group 17 17

Software SASOSs vs. Hardware Supported SASOSs • Software – Use stock 64 -bit RISC processors – Require additional software communication protocols for protection – Examples: Opal, Mungi, Grasshopper • Hardware supported – Require additional CPU hardware for protection – Examples: AS/400, Monads, Sombrero A HW-supported SASOS can take fuller advantage of single address space characteristics. 9/25/2020 ASU 64 -bit OS Group 18

SASOSs on Stock Processors vs. SASOSs with Additional CPU Protection Hardware SW SASOSs HW-Supported SASOSs System • Obtaining system services requires a • Obtaining operating system services is orthogonal to entry into privileged Services domain crossing AND entry into privileged supervisor mode • System architecture can be flat and Operating • Layered system structure - operating modular - OS services, environment System system kernel, executive services and servers and user-level programs Structure user-level servers layered on top of accessible to each other via ordinary each other procedure calls OOD • Direct support for OOD and OOP is • OOD and OOP can be directly and difficult - preprocessors and run-time supported - an object class can be OOP implemented directly as a protection systems are needed Support domain and a server as an instantiation of the class; base classes can be extended via user-level overrides 9/25/2020 ASU 64 -bit OS Group 19 19

SASOSs on Stock Processors vs. SASOSs with Additional CPU Protection Hardware (continued) SW SASOSs HW-Supported SASOSs • Copy set replication and Replication consistency management is usually Management page based and requires broadcasts reducing scalability Fault • Network-wide fault tolerance can Tolerance require additional mechanisms Object • Page-based object granularity Granularity Privilege Level for Basic Services Protection Policy and Implement ation 9/25/2020 • Access authorization, resource accounting and name services typically require kernel services or intervention • OS support for protection policy and protection implementation are intertwined 9/25/2020 • The common VA space enables simple scaleable per-object copy set replication and consistency management • Naturally provides a lowest level cache for fault tolerance • Object granularity can be independent of page granularity • Access authorization, resource accounting and name services can be done at user level • Clean separation of OS support for protection implementation and userlevel definition of protection policy is simple ASU 64 -bit OS Group 20 20

Motivation for a HW-supported SASOS Obtain • Improved program development environment • Greater performance • Better support for distributed applications By making fuller use of the properties of a single address space. 9/25/2020 ASU 64 -bit OS Group 21

to 9/25/2020 ASU 64 -bit OS Group 22

Sombrero Project Basic Premises • Economic and technological tradeoffs will increasingly favor single address space operation and SASOSs within ten years. • It is necessary to make changes to CPUresident protection and memory management hardware in order to design a SASOS that makes the paradigm shift viable. • This hardware is feasible now and so is a SASOS built on it. 9/25/2020 ASU 64 -bit OS Group 23

SOMBRERO DESIGN OVERVIEW • • • Sombrero Key Features Basic Abstractions Hardware Design Software Protection Data Structures Operating System Structure and System Architecture Distributed System Design 9/25/2020 ASU 64 -bit OS Group 24

Sombrero - Key Features • CPU-resident hardware protection – provides a substrate for simpler/faster protection domain and object operations • An open and extensible architecture that provides more choices for system structure • Network-wide transparent distribution of the single address space • Direct system level support for OOD and OOP • No TLB required - can be replaced by a single CAM-resident inverted page table at the memory bus 9/25/2020 ASU 64 -bit OS Group 25

BASIC ABSTRACTIONS • Memory Object Characterized by Range - a Distinct Variable Sized Contiguous Allocation of the Virtual Address Space - contains Memory Regions, sets of Virtual Addresses within a Range with Non-Overlapping Protection Attributes. • Protection Domain Set of Memory Objects and Protection Domains Reachable by a Thread executing within it and their Associated Access Permissions. There are General and Carrier Protection Domains (GPDs and CPDs). – GPD: Code, Data and other GPDs accessible to all threads within it – CPD: Data and GPDs privately accessible to an Individual Thread • Thread State of a Computation Represented by Current or most Relevant CPU Register State (PC, SP, General Registers, etc. ), Reachable Carrier Protection Domain Memory and Status Information. • Principal Unique Protected Identity that Represents a User or a System Service - binds Resources and Activities to an Owning User. 9/25/2020 4: 22: 20 PM ASU 64 -bit OS Group 26 26

HARDWARE DESIGN Sombrero RPLB Range Protection Lookaside Buffer • Functional Requirements • Logical Design • RPLB VLSI Synthesis 9/25/2020 ASU 64 -bit OS Group 27

FUNCTIONAL REQUIREMENTS • Separation of Address Translation and Address Protection Functions • Hardware Caching of Allowed Protection Domain Crossings • Protection Domains for Threads Distinct from the General Protection Context • Implicit Domain Crossing using Ordinary Instructions • Protection for Variable Granularity Object Sizes 9/25/2020 ASU 64 -bit OS Group 28

RPLB Logical Design • Ranges and Regions • Intra-Domain Operation • Cross-Domain Operation The RPLB stores co-located protection triples: <Accessing PD, Accessed Resource, access rights> where the PD and resource are represented by VAs 9/25/2020 ASU 64 -bit OS Group 29

RPLB LOGICAL DESIGN • A Range is a contiguous set of VAs • Unit of Protected Resource Access is Called a Region – Possibly noncontiguous set of VAs within a Range – Non-overlapping Protection Attributes from {rwxs} • A Range Contains one or more Regions 9/25/2020 The RPLB Stores Region Definitions • When Loaded with a Match Mask and a Don’t Care Mask, Internal Combinational Logic Produces a Range Mask that Defines a Region • The Figure shows the Definition of a Region consisting of 0 C 80 H - 0 CFFH and 0 D 80 H - 0 DFFH ASU 64 -bit OS Group 30

RPLB Intra-Domain Operation • On a Miss the RPLB is loaded with Protection Triples: <Accessing PD, Accessed VA Region, Access Rights> • These are Compared with CPU Emitted Data On Every Memory Reference to Determine whether the Access is Allowed. 9/25/2020 ASU 64 -bit OS Group 31

RPLB Cross-Domain Operation • On a Miss the RPLB is loaded with Protection Triples: <Accessing PD, Accessed PD Entry Point VA, Switch Access Right> + the New Protection Domain Virtual Address Name • These are Compared with CPU Emitted Data On Every Non-Local Memory Reference to Determine whether the Entry is Allowed. • For Allowed Accesses, the New GPD Name is Stored in the GPDBR and this is Followed by an Intra-Domain Access Attempt 9/25/2020 ASU 64 -bit OS Group 32

RPLB VLSI Synthesis • Proof of Concept - 1995 – VHDL Simulation – Mentor Design Tools – ASU ULSI Laboratory • Performance Estimate-1997 – Mentor Autologic II Synthesis Tool – Synopsis Design Compiler – Cascade Design Automation Static Timing Analyzer • Timing and Size Results – Timing - 8 -entry buffer • single location - 500 MHz • entire RPLB - 250 MHz – Size - 1. 27 mm 2 • Conclusion RPLB Architecture Synthesized with Synopsis Design Compiler Using 0. 35 -micron Library -- 500 MHz Pipelined Design Feasible 9/25/2020 ASU 64 -bit OS Group 33

Software Protection Data Structures • Control Blocks • Protection and Resource Access Lists (PRALs) – Protection domain Access Lists (PALs) – Resource Access Control Lists (RACLs) • PRAL Simulation 9/25/2020 ASU 64 -bit OS Group 34

Control Blocks • Memory Object Control Block (MOCB) • Protection Domain Control Block – General PD Control Block (GCB) – Carrier PD Control Block (TCB) • Access Descriptor (AD) • Principal Control Block (PCB) • Token Tracking Structure 9/25/2020 ASU 64 -bit OS Group 35

Protection and Resource Access Lists PRALs contain the data needed to convert between user policy information in the form of principals, resources and access rights and the CPU representation of these things in the form of protection domains, virtual addresses and access privileges. 9/25/2020 ASU 64 -bit OS Group 36

Resource Access Control Lists RACLs are classical Access Control Lists 9/25/2020 ASU 64 -bit OS Group 37

Protection Domain Access Lists PALs contain the same fundamental protection information held by classical capability lists 9/25/2020 ASU 64 -bit OS Group 38

PRAL Simulation • Compared performance of Sombrero RPLB + PRAL vs. Alpha NT TLB + Page Tables • Virtually everything had to be simulated Simulation Driver/RPLB/PRAL/TLB and Page Tables • Parameters: Number of Protection Domains (Threads) and Number and Size of Objects • Results – RPLB miss penalty on average 2 times TLB miss penalty – RPLB miss rate lower than TLB miss rate for many common scenarios (e. g. , objects greater than a page) – RPLB performance roughly comparable to TLB performance and better for larger objects 9/25/2020 ASU 64 -bit OS Group 39

Open and Extensible Architecture • Peer-level modular system structure: Executive provides a few basic services - user servers provide the rest. Operating system service methods can be overridden by user-defined function implementations. • Communication between application programs, user servers and executive services via ordinary procedure call and return. • OS services provided by instantiations of classes that are implemented directly as protection domains. • Services can be passive significantly reduced IPC costs • Pico-kernel (the CPU’s protection domain) handles redirections caused by HW privilege mode changes and a very few hardware-related operations. • Thread mobility and upcalls enable a cleaner separation of user policies and OS mechanisms 9/25/2020 ASU 64 -bit OS Group 40

Sombrero Architecture (a) Traditional View (b) Peer-Level Domain view 9/25/2020 ASU 64 -bit OS Group 41

Sombrero II Architecture Completely peer-level system structure • Kernel Services distributed among executive protection domains • No central kernel and no hardware protected kernel mode • A few Protection Domain Lock Registers name the protection domain that can access sensitive protected instructions and registers 9/25/2020 ASU 64 -bit OS Group 42

Open and Extensible Architecture (continued) • The hardware in a HW-supported SASOS can make use of the properties of a single virtual address namespace to support common referencing between programs. This frees the OS of having to perform this function and allows alternative system structures and ways to provide system services. 9/25/2020 ASU 64 -bit OS Group 43

Network-wide Transparent Distribution of the Single Address Space • Memory object, thread and protection domain migration supported by distributed surrogate kernel data structures enabled by tokens • Transparent and implicit consistency and concurrency policy - every VA is tied to a specific policy • Granularity of data transmission determined by policy • Copy-set management uses local data/minimizes broadcasts • Network routers use virtual addresses 9/25/2020 ASU 64 -bit OS Group 44

Distributed Object Copy Set Management Last Known Writer Graph Pruning of Modified Page Cache Graph Copy. Set Graph 9/25/2020 ASU 64 -bit OS Group 45

Direct System Level Support for Object-Oriented Design and Programming • An object class can be implemented directly as a protection domain and a server as an instantiation of the class; executive base classes can be extended via user overrides. • Servers can be passive - exported service entry points invoke methods. These are accessed with ordinary procedure call instructions. • Modular, peer level relationship between applications and user and system servers • Hardware protection provides encapsulation. 9/25/2020 ASU 64 -bit OS Group 46

Sombrero vs. SW- Supported SASOSs Opal and Mungi • Centralized kernel-resident data structure for protection triples vs. capabilities • Single inverted page table at the memory bus vs. multiple per-PD page tables • Carrier protection domain vs. proxy/guard and PD-extension for domain crossings • Direct support for object-oriented program development environment • Implicit PD crossing at EVERY level 9/25/2020 ASU 64 -bit OS Group 47

Sombrero vs. HW-Supported SASOSs AS/400 and Monads • Flat 64 -bit address space - no segments • No HW memory tagging or additional CPU instructions for capability and tag mgmt • Network-wide single address namespace • Single CAM-based inverted page table • Simple extensible executive • Availability of all single address space property advantages to applications 9/25/2020 ASU 64 -bit OS Group 48

SASOS Applications • Object-oriented distributed database management systems • Multiple database/multiple programming model applications, e. g. , CAD/CAM • Distributed parallel computations • Real-time systems 9/25/2020 ASU 64 -bit OS Group 49

‘Killer Ap’ To succeed a SASOS needs a Killer Application • Must benefit from SASOS advantages – common unique and permanent naming scheme – communications support for numerous large program modules – sharing and protection support for a huge distributed database • It must be widely used and currently costly Most likely candidate: Object-Oriented DDBMS 9/25/2020 ASU 64 -bit OS Group 50

Current Technological Trends that Favor a Hardware-Supported SASOS • Changing Tradeoffs • New and Improved Technologies 9/25/2020 ASU 64 -bit OS Group 51

Advantages from Changing Tradeoffs • Increasing costs and difficulties reducing CMOS chip feature sizes below 0. 1 micron • Increasing disparity between CPU cycle times and RAM and disk access times • Increasing chip real-estate • Decreasing ratio of inter-node access time to disk access time • Decreasing costs of storage 9/25/2020 ASU 64 -bit OS Group 52

Advantages from New and Improved Technologies • • • 9/25/2020 Multicast-oriented networks, e. g, ATM Huge local disks - 8 gigabytes Large local memories - 128 MB and above Large virtual first and second level caches Concurrent Multithreaded CPUs Widely distributed homogenous machines accessible over high BW low latency internet links and cable modems ASU 64 -bit OS Group 53

Sombrero Prototype Status • Prototype being developed on two Alpha 21164 boxes running NT • NT Alpha PALCode modified to specialize an NT process to include full address range • TLB misses outside normal address range forwarded to Sombrero extension of PALCode • Emulated RPLB triggered by TLB misses • Threads assigned to PDs within the specialized process • NT provides basic I/O and file facilities • First prototype, proof of concept and performance extrapolations expected before December 1998. 9/25/2020 ASU 64 -bit OS Group 54

Future Work (a. k. a. Things not completely worked out yet) • Implementation of a Universal Protection Domain – needed to reduce RPLB entries • Mini-System Call/Vectored Exception Mechanism for Sombrero II – needed for high speed executive protection domain communication 9/25/2020 ASU 64 -bit OS Group 55

Summary • Disadvantages of a HW-Supported SASOS To make the shift to the single address space paradigm worthwhile requires additional CPUresident HW. This causes: – Backwards HW and SW incompatibilities – Programming differences – Mastering new approaches to HW/SW design 9/25/2020 ASU 64 -bit OS Group 56

Summary • Advantages of a HW-Supported SASOS – Improved program development environment – Higher performance – Better support for distributed applications – A better match to the needs of real-time systems CPU-resident protection hardware and a SASOS that runs on it can be implemented now. This combination makes fuller use of the properties of a very large address space than contemporary process-oriented systems for both single node and distributed systems. 9/25/2020 ASU 64 -bit OS Group 57