Chapter 7 Securing Commercial Operating Systems Chapter Overview

Chapter 7 Securing Commercial Operating Systems

Chapter Overview • • • Retrofitting Security into a Commercial OS History of Retrofitting Commercial OS's Commercial Era Microkernel Era UNIX Era – – – • IX Domain and Type Enforcement Recent Unix Systems Summary

Retrofitting Security into a Commercial OS • Requires reference Monitor Concept – – – Complete Mediation Tamperproofing Verifiability

Complete Mediation Challenges • Identify all security-sensitive operations – – Some embedded deep inside the kernel code. Examples: • • • – Open Sockets Shared memory, etc. Covert channel identification is usually not even attempted

Tamperproofing Challenges • • Obvious: place in ring 0, but Kernel is often updated. /dev/kmem, /proc, Sysfs, netlink sockets → Every root process must STILL be part of the UNIX TCB

Verification Challenges • Must: – Mediation is implemented correctly, but • • – Large number of queries and processes. Complicate policies. Reference monitor implementation is correct • – Implemented in unsafe languages Policy enforces required security goals • • – Mediation interface designed manually Rest of TCB is huge. Rest of the TCB behaves correctly.

History of Retrofitting Commercial OS's • Commercial Era – – – • Retrofit security into OS → UNIX MLS Microkernel Era – – • Emulate system on security kernel Independent Server Processes → went to kernel New security models addressing both confidentiality and integrity Unix Era – Composed solutions from the two eras with focus on system integrity.

Commercial Era • Emulated Systems – – • • • Data Secure UNIX KSOS KVM/370 – 25% Performance overhead VAX/VMS DEC/Sandia Labs: MLS Secure Xenix (IBM) Access control and auditing – Added Compatibility • • – • Retrofitted Unix services Hidden subdirectories – Polyinstantiated file systems Trusted Path (Secure attention sequence) 1990 saw many secure Unix variants

Microkernel Era • Goal: minimal size kernel emphasizing system abstractions; no emphasis on security per se. • Source: Mach (1980's) – – Trusted Mach (Tmach) Distributed Trusted Mach (DTMach) Distributed Trusted OS (DTOS) Flask

Trusted Mach • • • Built by Trusted Information Systems (TIS) Added MLS for files, memory. Aim was to provide function for other systems like Unix and Windows. (Single level)

Distributed Trusted Mach • • Secure Computing Corporation and NSA Hybrid access control model: – – • MLS labels for confidentiality Type Enforcement labels for integrity (TE) Similar architecture to Tmach + servers for networking and general security policy server.

DTMach II • DTMach = Mach + security server – Security server = reference monitor outside the kernel • • – Each port access implies an authorization query For example, opening a file opens a port to the file server, etc. Security server used both MLS and TE rules. • TE rules: – – • code could only be modified by administrators Limited code that could be executed There were limitations: – For example, there was an arbitrary send right. . .

Distributed Trusted OS (DTOS) • AIM: True reference monitor in the Mach microkernel. • • Richer set of operations for ports than just send. Microkernel: – – • Managed labeling of subjects and kernel objects. Mediated each kernel operation by querying security server. Focus on verifiability of microkernel and TCB.

Flask • Fluke was a second generation microkernel developed at University of Utah, better than Mach. • • Flask = DTOS – Mach / Fluke More emphasis on TE.

UNIX Era • • • By early 1990's, many Unices had MLS support. Search for adding integrity (very ad-hoc at this point). Cover two systems: – – IX DTE

IX • • • AT&T prototype, enforces MLS and integrity. • Care has been taken to prevent tampering in the TCB. • • Includes a reference monitor over file access Mandatory access control policy providing both confidentiality and integrity protections. Verification not a goal. MLS was high water mark, for files and processes. However processes could not go beyond a certain ceiling.

IX (2) • • Integrity was Lo. Mac, with floors. • • No memory-mapped files. Since levels are dynamic, each data transfer must be checked/mediated. Trusted paths/pipes between processes (pex); a pex includes a label for the process at each end so that only that process may work with it.

An assured pipeline in IX

Domain and Type Enforcement • • • Trusted Information Systems: Problem: protecting TCB from vulnerable root processes Runs on Tmach system, but reference monitor added to OSF/1.

DTE Policy Model • • Subject types are now called Domains, object types are still types. • A domain describes how a process accesses files, signals other processes and creates processes. • DTE Unix defines limited protection domains for root processes. Key point is “least privilege”. • • Domain transitions are limited and their execution is limited also. Each domain is a triple (access rights to objects, access rights to subjects in other domains (signals), entry point program) Labeled Networking.

Recent Unix Systems • BSD variants – Trusted BSD • • • – – Reference monitor interface similar to LSM SEBSD is a version of SELinux for BSD Free. Bsd Jail Open. BSD emphasizes correct coding and configuration • • • – MAC, auditing, authentication Code separation Buffer overflow protection Least privilege configurations Net. BSD • • In-kernel authentication and verification of file execution Veriexec

Summary