Improving the Reliability of Commodity Operating Systems Introduction

Improving the Reliability of Commodity Operating Systems

Introduction l Nooks ¡ Allows existing OS extensions to execute safely in commodity kernels ¡ Use lightweight kernel protection domains Restricted write access to kernel memory l Track and validate all modifications to kernel data structures l

Motivation l Computer ¡ Cost l OS reliability a unsolved problem of failures continues to rise extensions have become prevalent ¡ 70% of Linux kernel code ¡ 35, 000 drivers on Windows XP ¡ Written by people who are less experienced in kernel organization

Motivation l Extensions ¡ In are leading cost of failures Windows XP, drivers cause 85% of failures ¡ In Linux, device drivers introduce 7 x errors than the rest of the kernel ¡ Extended OS cannot be tested completely

Nooks Approach l Target existing extension architecture l Use conventional C instead of type-safe languages l Aim to reduce the number of crashes due to drivers and extensions l Prototype implemented in Linux l Showed graceful recovery for 99% of fault injections

Related Work l Hardware approaches ¡ Capability-based l Recovery difficult for shared resources ¡ Segment l architectures Difficult to program l New OS structures ¡ Microkernels Good fault isolation l Rebooting required to restart services l

Related Work l Transaction-based ¡ Works systems well for file systems l Language-based ¡ Limited approaches applicability

Architecture l Core principles ¡ Design l for fault resistance, not fault tolerance Prevent and recover from most, not all ¡ Design for mistakes, not abuse Extensions are generally well-behaved (not malicious) l Can explore the design space between unproctected and safe l

Architecture l Implications + Can define an architecture that supports existing drivers with moderate performance costs - Malicious code can bypass these mechanisms

Goals l Isolation ¡ Need of kernel from extension failures to detect failures before they spread l Automatic recovery from failures l Backward compatibility

Functions l Reliability layer inserted between the extensions and the OS kernel ¡ Intercepts all interactions between the extensions and the OS kernel l Major functions ¡ Isolation ¡ Interposition ¡ Object tracking ¡ Recovery

Isolation l Lightweight kernel protection domain ¡ Write access to a limited portion of the kernel’s address space l Major tasks ¡ Creation, manipulation, and maintenance of lightweight kernel protection domains ¡ Inter-domain control transfer

Isolation l Extension procedure call (XPC) ¡ Similar to lightweight RPC ¡ Assume trusted interactions ¡ Asymmetric relationship l Kernel has more privileges

Interposition l The Nooks interposition mechanisms ¡ Make sure that All control flows between the kernel and extensions are through the XPC mechanism l All data flows between the kernel and extensions are managed by Nooks’ object-tracking code l l Extensions and the kernel communicate through wrapper stubs

Object Tracking l Maintains a list of kernel data structures that are manipulated by an extension l Controls all modifications to those structures l Provides object info for cleanup when an extension fails

Object Tracking l An object must be copied into an extension before it is modified l Object tracking code verifies the type and accessibility of each parameter being passed

Recovery l Nooks detects software faults ¡ When kernel services are invoked incorrectly ¡ When an extension consumes too many resources l Actions ¡ Return to the extension ¡ Generate an error code

Recovery l Nooks detects hardware faults ¡ Processor raises an exception during extension execution Attempts to read unmapped memory l Write memory outside of its protection domain l l. A user or a program trigger Nooks recovery explicitly

Recovery l Since extensions are decoupled from kernel, Nooks can freely release extension -held kernel structures, such as objects or locks, during the recovery process

Architecture

Implementation l Linux 2. 4. 18 ¡ Worst-case target ¡ 18 months of development ¡ 22, 000 lines of Nooks code (vs. 2. 4 million lines of Linux code and 50 million lines of Windows 2003 code)

Isolation l Two parts ¡ Memory management ¡ Extension procedure call

Memory Management l Kernel has read-write access to the entire address space l Each extension is restricted to read-only kernel access and read-write access to its local domain l Nooks maintains a copy of the kernel page table for each domain

Memory Management l Changing protection domains is not as costly as changing processes ¡ Protection domains share kernel address space

Extension Procedure Call l Transparent to both the kernel and its extensions l Managed by two functions ¡ nooks_driver_call(func_ptr, arg_list, domain) ¡ nooks_kernel_call(func_ptr, arg_list, domain) l Deferred ¡ Useful call mechanisms available for network drivers to queue up packets and perform bulk transfers

Changes to Linux Kernel l Maintain coherency between the kernel and extension page tables l Detect exceptions that occurs within Nooks’ protection domains l Locate tasks that are no longer collocated on the kernel stack due to isolation

Interposition l Provides wrapper stubs between extensions and the kernel ¡ Transparent l Kernel ¡ Make to the kernel and drivers modifications standard module load to bind extensions to wrappers instead of kernel functions ¡ The kernel is initialized to interpose on the Nooks’ call into extensions

Interposition l Some data references are interposed l Certain objects are linked directly into the extension for reading l Kernel modification calls are wrapped l Performance critical data structure ¡ Shadow object in extension that are synchronized before and after XPCs l Otherwise, just XPCs

Wrappers l Within the kernel’s protection domain l Three basic tasks ¡ Check parameters for validity ¡ Create a copy of kernel objects in the extension’s protection domain No serialization/deserialization necessary l Synchronization code placed in wrappers l ¡ Perform an XPC into the kernel or extension l Automatically generated

Wrapper Code Sharing l 50% of Nooks code base l Shared among multiple drivers

Object Tracking l Supports 43 kernel object types l Records the addresses of all objects in use by an extension l Records the association between the kernel and the extension versions of writable objects l Performs garbage collection l Determines whether to copy an object

Recovery l Recovery manager releases resources ¡ Unloading the extension ¡ Releasing its kernel and physical resources ¡ Reloading and restarting the extension l User-mode agent coordinates recovery l Each object is associated with a recovery function

Implementation Limitations l Nooks does not handle all possible errors ¡ Deliberate corruptions of system states ¡ Infinite loops l However, a moderate reduction of system crashes is a significant contribution

Achieving Transparency l Wrapper stubs for every call in the extension-kernel interface l Object-tracking code for every object type that is passed between the extension and the kernel l Nooks transparent to both the extension and the kernel

Reliability l Nooks can detect and recover 99% of extension faults

Test Methodology l Synthetic fault injection ¡ Automatically changes single instructions in the extension code to emulate common errors Uninitialized variables l Bad parameters l

Types of Extensions Isolated l Device drivers (network, sound cards) l Optional kernel subsystems (VFAT) l Application-specific kernel extension (k. HTTPd)

Test Environment l VMware ¡ Allows automation of crash testing without reboots l 5 extensions ¡ 400 tests each

Test Results l Not all faulty-injection trials cause faulty behavior

System Crashes l. A system crash is easiest to detect ¡ OS panics ¡ Hangs ¡ Reboots l Linux experienced 317 crashes l Nooks eliminated 313 crashes, or 99% l 4 deadlocks

System Crashes l Sound blaster and VFAT extensions are process-oriented ¡ Fewer crashes l k. HTTPd, pcnet 32, e 1000 are interrupted- based ¡ More crashes

Non-Fatal Extension Failures l Nooks cannot detect erroneous extension behaviors ¡ Network could disappear ¡ Mounted file system hangs

Recovery Errors l. A faulting extension is unloaded, reloaded, and restarted ¡ Works well with k. HTTPp ¡ Not as well with VFAT l Corruptions can propagate to disk if not detected in time

Summary of Reliability Experiments l Nooks eliminated 99% of the system crashes in extensions l Nooks eliminated nearly 60% of non-fatal extension failures

Performance l Dell 1. 7 GHz Pentium 4 l 890 MB of RAM l Sound. Blaster 16 l Intel Pro/1000 Gb Ethernet Adapter l 7200 RPM, 41 GB IDE HD l Linux 2. 4. 18

Sound Benchmark l Plays an MP 3 file at 128 Kb/sec l 150 XPCs/sec l Nooks imposes little overhead

Network Benchmark l netperformance tool l A node sends/receives a stream of 32 KB TCP messages via a 256 KB buffer ¡ 10% overhead

Compile Benchmark l Linux kernel compilation on VFAT l 25% slowdown

Web Server Benchmarks l httperf ¡ Repeatedly request a 1 -KB file and measure the maximum request rate ¡ 60% slowdown ¡ CPU bound l SPECweb 99 ¡ 3% slowdown

Summary l If the computation is not CPU bound, the penalty may not be important

Conclusions l Nooks is achievable with modest engineering effort l Extensions such as device drivers can be isolated without changes to extension code l Isolation and recovery can dramatically improve the system’s ability to survive extension faults