W 4118 Operating Systems Interrupt and System Call

W 4118 Operating Systems Interrupt and System Call in Linux Instructor: Junfeng Yang

Logistics q Find two teammates before next Thursday § Post ads in Course. Works

Last lecture q OS structure § § § Monolithic v. s. microkernel Modern OS: modules Virtual machine q Intro to Linux q Interrupts in Linux

Interrupts in Linux Memory Bus intr # IRQs PIC idtr INTR CPU 0 IDT intr # ISR How to handle interrupts? Mask points 255

Today q Interrupts in Linux (cont. ) § Interrupt handlers q System calls in Linux q Intro to Process

Nested Interrupts q q q What if a second interrupt occurs while an interrupt routine is executing? Generally a good thing to permit that — is it possible? And why is it a good thing?

Maximizing Parallelism q q q You want to keep all I/O devices as busy as possible In general, an I/O interrupt represents the end of an operation; another request should be issued as soon as possible Most devices don’t interfere with each others’ data structures; there’s no reason to block out other devices

Handling Nested Interrupts q q q Hardware invokes handler with interrupt disabled As soon as possible, unmask the global interrupt Interrupts from the same IRQ line? § q Wants to process in serial Thus, interrupt from same IRQ is not enabled during interrupt-handling

Interrupt Handling Philosophy q To preserve IRQ order on the same line, must disable incoming interrupts on same line § q q q New interrupts can get lost if controller buffer overflow Interrupt preempts what CPU was doing, which may be important Even not important, undesirable to block user program for long So, handler must run for a very short time! § Do as little as possible in the interrupt handler • Often just: queue a work item and set a flag § Defer non-critical actions till later

Intr handlers have no process context! q Interrupts (as opposed to exceptions) are not associated with particular instructions, nor the current process. It’s like an unexpected jump. • Why? Context switch expensive q Implication § § Interrupt handlers cannot call functions that may sleep (i. e. yield CPU to scheduler) ! Why not? • Scheduler only schedules processes, so wouldn’t know to reschedule the interrupt handler • The current process may be doing something dangerous, and cannot sleep

Linux Interrupt Handler Structure q Top half (th) and bottom half (bh) § § Top-half: do minimum work and return (ISR) Bottom-half: deferred processing (softirqs, tasklets, workqueues) Top half tasklet softirq workqueue Bottom half

Top half q q q Perform minimal, common functions: saving registers, unmasking other interrupts. Eventually, undoes that: restores registers, returns to previous context. Most important: call proper interrupt handler provided in device drivers (C program) Typically queue the request and set a flag for deferred processing Top half softirq Softirq flag = 10

Deferrable Work q q Three deferred work mechanisms: softirqs, tasklets, and work queues (tasklet built on top of softirq) All of these use request queues § q Think of requests as (function, args) All can be interrupted Top half tasklet softirq workqueue Bottom half

Softirqs q q Types are statically allocated: at kernel compile time Limited number: Priority Type 0 High-priority tasklets (generic) 1 Timer interrupts 2 Network transmission 3 Network reception 4 SCSI disks 5 Regular tasklets (generic) q Singled out timer, net and scsi disk because these are the most important for server performance q Each type mapped to a bit in a per-CPU bitmask. To raise a softirq = simply set bit q What does a softirq handler do?

Example: Network card Soft. IRQ q Parse packet header q Verify checksum q Deliver packet up to the network stack q return Linux-2. 6. 11/net/core/dev. c, function net_rx_action

Running Softirqs q When to execute softirq? § § § q Run at various points by the kernel, using current process’s context Most important: after handling IRQs and after timer interrupts Essentially polling Problem: while processing one softirq, another is raised. Process it? § § No long delay for new irq Always starve user program when long softirq burst • Livelock!

Livelock q q 100% CPU utilization, but no progress Why? § Need user program to eventually process requests • E. g. webserver § However, if too many interrupt requests, starve user program § Big deal for networking in 90 s § Solution: • Eliminating receive livelock in an interrupt-driven kernel, Jeffrey C. Mogul, K. K. Ramakrishnan • Adopted into Linux

Avoid Livelock q q Goal: provide user program fair share of CPU time despite interrupt burst Quota + dedicated context ksoftirqd § Process up to N softirqs for one softirq hanlder invocation • Bound time spent in handler § Process the rest in ksoftirqd • ksoftirqd subject to scheduling, as user process • Provide fairness to user process

Tasklets q Problem: softirq is static § q Solution: tasklets § § § q To add a new type of Softirq, need to convince Linus! Built on top of softirq New types are created and destroyed dynamically Simplified for muliticore processing: at any time, only one tasklet among all of the same type can run Problem with softirq and tasklets: they have no process contexts either, thus cannot sleep

Work Queues q q Softirqs and tasklets run in an interrupt context; work queues have a process context The idea: § § § q You throw work (fn, args) to a workqueue Workqueue add to an internal FIFO queue A dedicated workqueue process loops forever, dequeuing (fn, args), and running fn(args) Because they have a process context, they can sleep

Monitoring Interrupt Activity q q Linux has a pseudo-file system, /proc, for monitoring (and sometimes changing) kernel behavior Run cat /proc/interrupts to see what’s going on

CPU 0 0: 162 IO-APIC-edge timer 1: 0 IO-APIC-edge i 8042 4: 10 IO-APIC-edge 7: 0 IO-APIC-edge 8: 1232299 IO-APIC-edge parport 0 rtc 9: 0 IO-APIC-fasteoi acpi 12: 1 IO-APIC-edge i 8042 16: 19256781 IO-APIC-fasteoi uhci_hcd: usb 1, … 17: 79 IO-APIC-fasteoi uhci_hcd: usb 2, uhci_hcd: usb 4 … # Columns: IRQ, count, interrupt controller, devices

Today q Interrupts in Linux (cont. ) q System calls in Linux q Intro to Process

API – System Call – OS Relationship { printf(“hello world!n”); } libc User mode %eax = sys_write; int 0 x 80 system_call() { fn = syscalls[%eax] kernel mode 0 x 80 IDT } syscalls table sys_write(…) { // do real work }

System Calls vs. Library Calls q Library calls are much faster than system calls § § § q If you can do it in user space, you should strlen? write? Learn what a library call/system call do: § § § Documents are called “manpages, ” divided into sections Library calls (section 3) e. g. man 3 strlen System calls (section 2) e. g. man 2 write

Next: Syscall Wrapper Macros { printf(“hello world!n”); } libc User mode %eax = sys_write; int 0 x 80 system_call() { fn = syscalls[%eax] kernel mode 0 x 80 IDT } syscalls table sys_write(…) { // do real work }

Syscall Wrapper Macros q q Generating the assembly code for trapping into the kernel is complex so Linux provides a set of macros to do this for you! Macros with name _syscall. N(), where N is the number of system call parameters § § § q q _syscall. N(return_type, name, arg 1 type, arg 1 name, …) in linux-2. 6. 11/include/asm-i 386/unistd. h Macro will expands to a wrapper function Example: § long open(const char *filename, int flags, int mode); § _syscall 3(long, open, const char *, filename, int, flags, int, mode) NOTE: _syscall. N obsolete after 2. 6. 18; now syscall (…), can take different # of args

Lib call/Syscall Return Codes q q Library calls return -1 on error and place a specific error code in the global variable errno System calls return specific negative values to indicate an error § q Most system calls return -errno The library wrapper code is responsible for conforming the return values to the errno convention

Next: Syscall Implementation { printf(“hello world!n”); } libc User mode %eax = sys_write; int 0 x 80 system_call() { fn = syscalls[%eax] kernel mode 0 x 80 IDT } syscalls table sys_write(…) { // do real work }

System call handler. section. text system_call: // copy parameters from registers onto stack… call sys_call_table(, %eax, 4) jmp ret_from_sys_call: // perform rescheduling and signal-handling… iret // return to caller (in user-mode) // File arch/i 386/kernel/entry. S Why jump table? Can’t we use if-then-else?

The system-call jump-table q q q There approximately 300 system-calls Any specific system-call is selected by its IDnumber (it’s placed into register %eax) It would be inefficient to use if-else tests or even a switch-statement to transfer to the service-routine’s entry-point Instead an array of function-pointers is directly accessed (using the ID-number) This array is named ‘sys_call_table[]’ § Defined in file arch/i 386/kernel/entry. S

System call table definition. section. data sys_call_table: . long sys_restart_syscall. long sys_exit. long sys_fork. long sys_read. long sys_write … NOTE: syscall numbers cannot be reused (why? ); deprecated syscalls are implemented by a special “not implemented” syscall (sys_ni_syscall)

Syscall Naming Convention q q q Usually a library function “foo()” will do some work and then call a system call (“sys_foo()”) In Linux, all system calls begin with “sys_” Often “sys_foo()” just does some simple error checking and then calls a worker function named “do_foo()”

Tracing System Calls q q q Linux has a powerful mechanism for tracing system call execution for a compiled application Output is printed for each system call as it is executed, including parameters and return codes The ptrace() system call is used § q q Also used by debuggers (breakpoint, singlestep, etc) Use the “strace” command (man strace for info) You can trace library calls using the “ltrace” command

Passing system call parameters q q The first parameter is always the syscall # § eax on Intel Linux allows up to six additional parameters § ebx, ecx, edx, esi, edi, ebp on Intel System calls that require more parameters package the remaining params in a struct and pass a pointer to that struct as the sixth parameter Problem: must validate pointers § § Could be invalid, e. g. NULL crash OS Or worse, could point to OS, device memory security hole

How to validate user pointers? q Too expensive to do a thorough check § q Need to check that the pointer is within all valid memory regions of the calling process Solution: No comprehensive check § § Linux does a simple check for address pointers and only determines if pointer variables are within the largest possible range of user memory (more details when talking about process) Even if a pointer value passes this check, it is still quite possible that the specific value is invalid Dereferencing an invalid pointer in kernel code would normally be interpreted as a kernel bug and generate an Oops message on the console and kill the offending process Linux does something very sophisticated to avoid this situation

Handling faults due to user-pointers q Kernel code must access user-pointers using a small set of “paranoid” routines (e. g. copy_from_user) § q q q Thus, kernel knows what addresses in its code can throw invalid memory access exceptions (page fault) When a page fault occurs, the kernel’s page fault handler checks the faulting EIP (recall: saved by hw) If EIP matches one of the paranoid routines, kernel will not oops; instead, will call “fixup” code Many violations of this rule in Linux. Once built a checker and found tons of security holes

Paranoid functions to access user pointers Function Action get_user(), __get_user() reads integer (1, 2, 4 bytes) put_user(), __put_user() writes integer (1, 2, 4 bytes) copy_from_user(), __copy_from_user copy a block from user space copy_to_user(), __copy_to_user() copy a block to user space strncpy_from_user(), __strncpy_from_user() copies null-terminated string from user space strnlen_user(), __strnlen_user() returns length of null-terminated string in user space clear_user(), __clear_user() fills memory area with zeros

How to find “fixup” code? q Exception table § q q q Faulting instruction address fixup code On page fault, kernel scans exception table to find the fixup code Typically the fixup code terminates the system call with an EINVAL error code (means: invalid arguments) Some ELF tricks help to generate exception table and implement fixup code; see ULK Chapter 10 for gruesome details

Intel Fast System Calls q q q int 0 x 80 not used any more (I lied …) Intel has a hardware optimization (sysenter) that provides an optimized system call invocation Read the gory details in ULK Chapter 10

Today q Interrupts in Linux (cont. ) q System calls in Linux q Intro to Process § § What are processes? Why need them?

What is a Process q q q “Program in execution” “virtual CPU” Process is an execution stream in the context of a particular process state Execution stream: a stream of instructions § § Running piece of code sequential sequence of instructions

What is a Process? (cont. ) q Process state: determines the effects of the instructions. § Stuff that the running code can affect or be affected by Registers § Memory: everything a process can address § I/O § More … § • General purpose, floating point, EIP … • Code, data, stack, heap • File descriptor table stack reg SP heap IP data cpu code mem

Program v. s. Process q Process != program § § q Program: static code and static data Process: dynamic instantiation of code and data Process <-> program: no 1: 1 mapping § Process > program: code + data + other things program process main() { f(x); stack for f() regs } } f(int x) { IP } } heap

Program v. s. Process (cont. ) q Process <-> program: no 1: 1 mapping § Program > process: one program can invoke multiple processes • E. g. shell can run commands in different processes § Process > program: can have multiple processes of the same program • E. g. Multiple users run multiple /usr/bin/tcsh

Address Space (AS) q q q More details when discussing memory management AS = All memory a process can address + addresses Virtual address space: § § Really large memory to use Linear array of bytes: [0, N), N roughly 2^32, 2^64 q Process and virtual address space: 1 mapping q Key: an AS is a protection domain § One process can’t address another process’s address space (without permission) • E. g. Value stored at 0 x 800 abcd in p 1 is different from 0 x 800 abcd § Thus can’t read/write

Process v. s. Threads q q More details when discussing threads Process != Threads: many streams of executions in one process Threads share address space threads process main() { f(x); } stack for f() regs main() { f(x); } f(int x) { IP f(int x) { } heap } stack for f() regs IP IP heap

Why need processes? q Divide and conquer § q Sequential § q Decompose a large problem into smaller ones easier to think well contained smaller problems Easier to think about Increase performance. § System has many concurrent jobs going on

System categorization q Most OS support process q Uniprogramming: only one process at a time § § q Good: simple Bad: low utilization, low interactivity Multiprogramming: multiple at a time § § When one proc blocks (e. g. I/O), switch to another NOTE: different from multiprocessing (systems with multiple processors) Good: increase utilization, interactivity Bad: complex

Multiprogramming q OS support for multiprogramming § § Policy: scheduling, what proc to run? (next week) Mechanism: • dispatching, how to run/block process? • how to protect from one another? q Separation of policy and mechanism § § Recurring theme in OS Policy: decision making with some performance metric and workload • Scheduling (next week) § Mechanism: low-level code to implement decisions • Dispatching (today)

Process state diagram q Process state § § § New: being created Running: instructions are running on CPU Waiting: waiting for some event (e. g. IO) Ready: waiting to be assigned a CPU Terminated: finished

Process dispatching mechanism q q OS stores processes on system-wide lists OS dispatching loop: while(1) { proc = choose(ready procs); load proc state; run proc for a while; save proc state; } Q 1: how to gain control? Q 2: what state must be saved?

Backup slides

How to gain control? q q Cooperative v. s. preemptive Cooperative. Process voluntarily yield control to OS § § When? System call Why bad? OS trusts process ! • Malicious process? Bugs? q Preemptive § § When? Interrupt, especially timer (studied before) OS trusts no one !

What state must be saved? q Registers Memory? I/O? q Context switch q q § § § How? Why hard? Need code to save registers, but code needs registers Why expensive? A lot of registers to load and store

Interrupt Return Code Path q Interleaved assembly entry points: § § q ret_from_exception() ret_from_intr() ret_from_sys_call() ret_from_fork() Things that happen: § § Run scheduler if necessary Return to user mode if no nested handlers • Restore context, user-stack, switch mode • Re-enable interrupts if necessary Deliver pending signals (Some DOS emulation stuff – VM 86 Mode)

Interrupt Handling Portability q q q Which has a higher priority, a disk interrupt or a network interrupt? Different CPU architectures make different decisions By not assuming or enforcing any priority, Linux becomes more portable

Interrupt Stack q When an interrupt occurs, what stack is used? § § § q Exceptions: The kernel stack of the current process, whatever it is, is used (There’s always some process running — the “idle” process, if nothing else) Interrupts: hard IRQ stack (1 per processor) Soft. IRQs: soft IRQ stack (1 per processor) These stacks are configured in the IDT and TSS at boot time by the kernel

Invoking System Calls user-mode (restricted privileges) … xyz() … app making system call wrapper routine in std C library kernel-mode (unrestricted privileges) sys_xyz() { … } ret xyz { … int 0 x 80; … } call int 0 x 80 iret system call service routine ret system_call: … sys_xyz(); … system call handler

The ‘jump-table’ idea sys_call_table 0 sys_restart_syscall 1 sys_exit 2 sys_fork 3 sys_read 4 sys_write 5 sys_open 6 7 8 sys_close …etc… . section. text

Mode vs. Space q q q q Recall that an address space is the collection of valid addresses that a process can reference at any given time (0 … 4 GB on a 32 bit processor) The kernel is a logically separate address space (separate code/data) from all user processes Trapping to the kernel logically involves changing the address space (like a process context switch) Modern OSes use a trick to optimize system calls: every process gets at most 3 GB of virtual addresses (0. . 3 GB) and a copy of the kernel address space is mapped into *every* process address space (3. . 4 GB) The kernel code is mapped but not accessible in user-mode so processes can only “see” the kernel code when they trap into the kernel and the mode bit is changed Pros: save TLB flushes, make copying in/out of user space easy Cons: steals processor address space, limited kernel mapping

Process Address Space 4 GB Privilege-level 0 Kernel space 3 GB User-mode stack-area User space Privilege-level 3 Shared runtime-libraries Task’s code and data 0 GB kernel-mode stack

Syscall Wrapper Macros q q q Generating the assembly code for trapping into the kernel is complex so Linux provides a set of macros to do this for you! There are seven macros with names _syscall. N() where N is a digit between 0 and 6 and N corresponds to the number of parameters The macros are a bit strange and accept data types as parameters For each macro there are 2 + 2*N parameters; the first two correspond to the return type of syscall (usually long) and the syscall name; the remaining 2*N parameters are the type and name of each syscall parameter Example: § § long open(const char *filename, int flags, int mode); _syscall 3(long, open, const char *, filename, int, flags, int, mode)

Linux Interrupt Summary q Types of interrupts q Device IRQs APICs Interrupt q ISR q Soft. IRQs, tasklets, workqueues, ksoftirqd § § q Interrupt context vs. process context Who can block Nested interrupt

Blocking System Calls q q q System calls often block in the kernel (e. g. waiting for IO completion) When a syscall blocks, the scheduler is called and selects another process to run Linux distinguishes “slow” and “fast” syscalls: § § q Slow: may block indefinitely (e. g. network read) Fast: should eventually return (e. g. disk read) Slow syscalls can be “interrupted” by the delivery of a signal (e. g. Control-C)

Nested Execution q Interrupts can be interrupted § § q Exceptions can be interrupted § q By different interrupts; handlers need not be reentrant No notion of priority in Linux Small portions execute with interrupts disabled Interrupts remain pending until acked by CPU By interrupts (devices needing service) Exceptions can nest two levels deep § § § Exceptions indicate coding error Exception code (kernel code) shouldn’t have bugs Page fault is possible (trying to touch user data)

Example: Network ISR (receive) q if NIC has incoming packets § § q Device already copied packet to an OS buffer ISR allocates a descriptor (sk_buff) for this buffer ISR queue this descriptor raises a softirq (== set a flag, explained next) return (will enable interrupt)

More: lspci $ cat /proc/pci PCI devices found: Bus 0, device 0, function 0: Host bridge: PCI device 8086: 2550 (Intel Corp. ) (rev 3). Prefetchable 32 bit memory at 0 xe 8000000 [0 xebffffff]. Bus 0, device 29, function 1: USB Controller: Intel Corp. 82801 DB USB (Hub #2) (rev 2). IRQ 19. I/O at 0 xd 400 [0 xd 41 f]. Bus 0, device 31, function 1: IDE interface: Intel Corp. 82801 DB ICH 4 IDE (rev 2). IRQ 16. I/O at 0 xf 000 [0 xf 00 f]. Non-prefetchable 32 bit memory at 0 x 80000000 [0 x 800003 ff]. Bus 3, device 1, function 0: Ethernet controller: Broadcom Net. Xtreme BCM 5703 X Gigabit Eth (rev 2). IRQ 48. Master Capable. Latency=64. Min Gnt=64. Non-prefetchable 64 bit memory at 0 xf 7000000 [0 xf 700 ffff].