Chapter 7 Securing Commercial Operating Systems Chapter Overview

Chapter 7 Securing Commercial Operating Systems

Chapter Overview • Retrofitting Security into a Commercial OS • History of Retrofitting Commercial OS's • Commercial Era • Microkernel Era • UNIX Era • – IX – Domain and Type Enforcement – Recent Unix Systems Summary

Retrofitting Security into a Commercial OS • Requires reference Monitor Concept – – – Complete Mediation Tamperproofing Verifiability

Complete Mediation Challenges • Identify all security-sensitive operations – Some embedded deep inside the kernel code. – Examples: – • Open • Sockets • Shared memory, etc. Covert channel identification is usually not even attempted

Tamperproofing Challenges • Obvious: place in ring 0, but • Kernel is often updated. • /dev/kmem, /proc, Sysfs, netlink sockets • → Every root process must STILL be part of the UNIX TCB

Verification Challenges • Musts: – Mediation is implemented correctly, but – • Mediation interface designed manually • Implemented in unsafe languages Policy enforces required security goals – • Large number of queries and processes. • Complicate policies. Reference monitor implementation is correct – • Rest of TCB is huge. Rest of the TCB behaves correctly.

History of Retrofitting Commercial OS's • • • Commercial Era – Emulate system on security kernel – Retrofit security into OS – → UNIX MLS Microkernel Era – Independent Server Processes → went to kernel – New security models addressing both confidentiality and integrity Unix Era – Composed solutions from the two eras with focus on system integrity.

Commercial Era • Emulated Systems – Data Secure UNIX – KSOS • KVM/370 – 25% Performance overhead • VAX/VMS DEC/Sandia Labs: MLS • Secure Xenix (IBM) Access control and auditing – – • Added Compatibility • Retrofitted Unix services • Hidden subdirectories – Polyinstantiated file systems Trusted Path (Secure attention sequence) 1990 saw many secure Unix variants

Microkernel Era • Goal: minimal size kernel emphasizing system abstractions; no emphasis on security per se. • Source: Mach (1980's) – – Trusted Mach (Tmach) Distributed Trusted Mach (DTMach) Distributed Trusted OS (DTOS) Flask

Trusted Mach • Built by Trusted Information Systems (TIS) • Added MLS for files, memory. • Aim was to provide function for other systems like Unix and Windows. (Single level)

Distributed Trusted Mach • Secure Computing Corporation and NSA • Hybrid access control model: – – • MLS labels for confidentiality Type Enforcement labels for integrity (TE) Similar architecture to Tmach + servers for networking and general security policy server.

DTMach II • DTMach = Mach + security server – – Security server = reference monitor outside the kernel • Each port access implies an authorization query • For example, opening a file opens a port to the file server, etc. Security server used both MLS and TE rules. • TE rules: • – code could only be modified by administrators – Limited code that could be executed There were limitations: – For example, there was an arbitrary send right. . .

Distributed Trusted OS (DTOS) • AIM: True reference monitor in the Mach microkernel. • Richer set of operations for ports than just send. • Microkernel: • – Managed labeling of subjects and kernel objects. – Mediated each kernel operation by querying security server. Focus on verifiability of microkernel and TCB.

Flask • Fluke was a second generation microkernel developed at University of Utah, better than Mach. • Flask = DTOS – Mach / Fluke • More emphasis on TE.

UNIX Era • By early 1990's, many Unices had MLS support. • Search for adding integrity (very ad-hoc at this point). • Cover two systems: – – IX DTE

IX • AT&T prototype, enforces MLS and integrity. • Includes a reference monitor over file access • Mandatory access control policy providing both confidentiality and integrity protections. • Care has been taken to prevent tampering in the TCB. • Verification not a goal. • MLS was high water mark, for files and processes. However processes could not go beyond a certain ceiling.

IX (2) • Integrity was Lo. Mac, with floors. • Since levels are dynamic, each data transfer must be checked/mediated. • No memory-mapped files. • Trusted paths/pipes between processes (pex); a pex includes a label for the process at each end so that only that process may work with it.

An assured pipeline in IX

Domain and Type Enforcement • Trusted Information Systems: • Problem: protecting TCB from vulnerable root processes • Runs on Tmach system, but reference monitor added to OSF/1.

DTE Policy Model • Subject types are now called Domains, object types are still types. • Each domain is a triple (access rights to objects, access rights to subjects in other domains (signals), entry point program) • A domain describes how a process accesses files, signals other processes and creates processes. • DTE Unix defines limited protection domains for root processes. Key point is “least privilege”. • Domain transitions are limited and their execution is limited also. • Labeled Networking.

Recent Unix Systems • BSD variants – Trusted BSD • MAC, auditing, authentication • Reference monitor interface similar to LSM • SEBSD is a version of SELinux for BSD – Free. Bsd Jail – Open. BSD emphasizes correct coding and configuration – • Code separation • Buffer overflow protection • Least privilege configurations Net. BSD • In-kernel authentication and verification of file execution • Veriexec

Summary Retrofitting Security into a Commercial OS - Requirements and Challenges - History • Commercial Era • Microkernel Era • Unix Era – recent Unix variants