Week 7 System Calls Kernel Threads Kernel Debugging

Week 7 System Calls, Kernel Threads, Kernel Debugging Sarah Diesburg Florida State University 1

First… n Any questions on q q Part 1 – 5 system calls Part 2 – xtime proc module 2

Story of Kernel Development Some context… 3

In the old days… n n There were no modules or virtual machines The kernel is a program q q Has code, can compile, re-compile, make executable When changes needed to be made, developers make changes in source and re-compile 4

How is the kernel different from a regular program? n Mostly in how it is executed q q q n Boot loader loads the kernel image/executable during boot time Sets kernel mode Jumps to the entry point in the image/executable Remember the generic booting sequence? 5

Quick Question n How would you make changes to the kernel and run those changes? 1. 2. 3. 4. 5. Make changes to the source Re-complie the kernel source Re-install the kernel source Make sure the bootloader sees the new kernel image (grub) Reboot and profit! 6

Getting more modern. . n n Modules were created as bits of code that can be loaded and unloaded by the kernel in kernel mode Made development easier q Instead of re-compiling, re-installing, and rebooting into the new kernel, one could just recompile and load a module 7

Quick Question n How would you make changes to a module and run those changes? 1. 2. 3. Make changes to module source code Re-compile the module Load the new module 8

Present Day n n Reboots into new kernels and loading new modules often freezes machines Enter virtual machine software q q Process that emulates the hardware necessary to run an OS in user-space Guest OS is executed inside the virtual machine process! 9

New System Calls Fun but tricky! 10

Implementing System Calls int start_elevator(void); int issue_request(int #1, int #2, int #3); int stop_elevator(void); n Need to implement the functions above. But how?

Adding a System Call to Kernel Files to add: n q q q LINUX_DIR/PROJECT_NAME/Makefile LINUX_DIR/PROJECT_NAME. c LINUX_DIR/PROJECT_NAME/NEW_SYSCALLS. c Files to modify: n q q LINUX_DIR/arch/x 86/kernel/syscall_table_32. S LINUX_DIR/include/asm-generic/unistd. h LINUX_DIR/include/linux/syscalls. h LINUX_DIR/Makefile

Sample System Call n Let’s add a sample module to the kernel that defines a sample system call n test_newsyscall(int test_int); q q n Takes an int test_int and issues a printk on it Returns test_int Problem – If our module isn’t loaded, what happens if we call our sample system call? 13

Project 2 System Call Model Elevator module Core kernel User program 14

Project 2 System Call Model Elevator module Core kernel User issues system call, core kernel looks up system call in system call table User program 15

Project 2 System Call Model Elevator module performs system call action Elevator module Core kernel User program 16

Project 2 System Call Model Elevator module returns result of system call Elevator module Core kernel User program 17

Project 2 System Call Model Elevator module Core kernel forwards result of system call to user program User program 18

What happens if elevator module is not loaded? Core kernel User program 19

What happens if elevator module is not loaded? Core kernel User issues system call, core kernel looks up system call in system call table User program 20

What happens if elevator module is not loaded? Elevator module is not loaded to perform the action…. OOPS! Core kernel User program 21

Module System Calls n We must create a wrapper system call! q q Wrapper will call module function if module loaded, else returns an error Must be created in a separate, built-in kernel file in the project folder 22

Function Pointers n We will implement our system call wrapper with a function pointer q n Pointer to a function Function pointer can point to any function that you implement that q q Takes the same input variable types Returns the same return type 23

Function Pointers long (*STUB_test_newsyscall)(int test_int) = NULL; n Function pointer that q q n Returns a long Name is STUB_test_newsyscall Takes parameter int test_int Function pointer set to NULL Can set function pointer to a local function you implement 24

Elevator Project n Create a file in your elevator project that just contains the system call information q KERNEL_DIR/PROJECT_DIR/newsyscalls. c 25

KERNEL_DIR/PROJECT_DIR/ newsyscalls. c #include <linux/linkage. h> #include <linux/kernel. h> #include <linux/module. h> /* System call stub. We initialize the stub function to be NULL. */ long (*STUB_test_newsyscall)(int test_int) = NULL; EXPORT_SYMBOL(STUB_test_newsyscall); /* System call wrapper. If the stub is not NULL, it will be run, otherwise returns -ENOSYS */ asmlinkage long sys_test_newsyscall(int test_int) { if (STUB_test_newsyscall) return STUB_test_newsyscall(test_int) else return -ENOSYS; 26 }

KERNEL_DIR/PROJECT_DIR/ newsyscalls. c #include <linux/linkage. h> #include <linux/kernel. h> #include <linux/module. h> Function pointer /* System call stub. We initialize the stub function to be NULL. */ long (*STUB_test_newsyscall)(int test_int) = NULL; EXPORT_SYMBOL(STUB_test_newsyscall); /* System call wrapper. If the stub is not NULL, it will be run, otherwise returns -ENOSYS */ asmlinkage long sys_test_newsyscall(int test_int) { if (STUB_test_newsyscall) return STUB_test_newsyscall(test_int) else return -ENOSYS; 27 }

KERNEL_DIR/PROJECT_DIR/ newsyscalls. c #include <linux/linkage. h> #include <linux/kernel. h> #include <linux/module. h> Export the pointer so we can access /* System call stub. We initialize the stub function to be NULL. it later */ long (*STUB_test_newsyscall)(int test_int) = NULL; EXPORT_SYMBOL(STUB_test_newsyscall); /* System call wrapper. If the stub is not NULL, it will be run, otherwise returns -ENOSYS */ asmlinkage long sys_test_newsyscall(int test_int) { if (STUB_test_newsyscall) return STUB_test_newsyscall(test_int) else return -ENOSYS; 28 }

KERNEL_DIR/PROJECT_DIR/ newsyscalls. c #include <linux/linkage. h> #include <linux/kernel. h> #include <linux/module. h> /* System call stub. We initialize the stub function to be NULL. */ long (*STUB_test_newsyscall)(int test_int) = NULL; EXPORT_SYMBOL(STUB_test_newsyscall); /* System call wrapper. If the stub is not NULL, it will be run, System call otherwise returns -ENOSYS */ wrapper asmlinkage long sys_test_newsyscall(int test_int) { if (STUB_test_newsyscall) return STUB_test_newsyscall(test_int) else return -ENOSYS; 29 }

Elevator Project n Next create a separate file that q q q Holds your module code Registers the system call pointer Actually implements the system call behavior 30

Inside KERNEL_DIR/PROJECT_DIR/ PROJECT_NAME. C /* Extern system call stub declarations */ extern long (*STUB_test_newsyscall)(int test_int); long my_test_newsyscall(int test) { printk("%s: Your int is %in", __FUNCTION__, test); return test; } my_module_init() { STUB_test_newsyscall=&(my_test_newsyscall); return 0; } my_module_exit() { STUB_test_newsyscall=NULL; } 31

Inside KERNEL_DIR/PROJECT_DIR/ PROJECT_NAME. C /* Extern system call stub declarations */ extern long (*STUB_test_newsyscall)(int test_int); Gain access to stub function pointer. long my_test_newsyscall(int test) { printk("%s: Your int is %in", __FUNCTION__, test); return test; } my_module_init() { STUB_test_newsyscall=&(my_test_newsyscall); return 0; } my_module_exit() { STUB_test_newsyscall=NULL; } 32

Inside KERNEL_DIR/PROJECT_DIR/ PROJECT_NAME. C /* Extern system call stub declarations */ extern long (*STUB_test_newsyscall)(int test_int); Local function that implements syscall long my_test_newsyscall(int test) { printk("%s: Your int is %in", __FUNCTION__, test); return test; } my_module_init() { STUB_test_newsyscall=&(my_test_newsyscall); return 0; } my_module_exit() { STUB_test_newsyscall=NULL; } 33

Inside KERNEL_DIR/PROJECT_DIR/ PROJECT_NAME. C /* Extern system call stub declarations */ extern long (*STUB_test_newsyscall)(int test_int); long my_test_newsyscall(int test) { printk("%s: Your int is %in", __FUNCTION__, test); return test; } Set stub function pointer to local function in init my_module_init() { STUB_test_newsyscall=&(my_test_newsyscall); return 0; } my_module_exit() { STUB_test_newsyscall=NULL; } 34

Inside KERNEL_DIR/PROJECT_DIR/ PROJECT_NAME. C /* Extern system call stub declarations */ extern long (*STUB_test_newsyscall)(int test_int); long my_test_newsyscall(int test) { printk("%s: Your int is %in", __FUNCTION__, test); return test; } my_module_init() { STUB_test_newsyscall=&(my_test_newsyscall); return 0; Reset stub } my_module_exit() { STUB_test_newsyscall=NULL; } function pointer to NULL on module unload 35

KERNEL_DIR/PROJECT_DIR/M akefile obj-m : = my_module. o obj-y : = newsyscalls. o KDIR : = /lib/modules/2. 6. 32/build PWD : = $(shell pwd) default: $(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules 36

KERNEL_DIR/PROJECT_DIR/M akefile obj-m : = my_module. o obj-y : = newsyscalls. o Compile as a module KDIR : = /lib/modules/2. 6. 32/build PWD : = $(shell pwd) default: $(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules 37

KERNEL_DIR/PROJECT_DIR/M akefile obj-m : = my_module. o obj-y : = newsyscalls. o Compile as kernel built-in KDIR : = /lib/modules/2. 6. 32/build PWD : = $(shell pwd) default: $(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules 38

Core Kernel Additions n Add the new system call to the core kernel system call table q n Modify three files Add the project directory to the main Makefile q Modify one file 39

Modifying syscall_table_32. S …. long sys_preadv sys_pwritev sys_rt_tgsigqueueinfo /* 335 */ sys_perf_event_open

Modifying syscall_table_32. S …. long sys_preadv sys_pwritev sys_rt_tgsigqueueinfo /* 335 */ sys_perf_event_open sys_test_newsyscall /* 337 */ § Add new system call to the end of the file. § Remember the number – you’ll need it in userspace!

Modifying unistd. h /* midfile */ #define __NR_perf_event_open 241 __SYSCALL(__NR_perf_event_open, sys_perf_event_open) #undef __NR_syscalls #define __NR_syscalls 242 /* midfile */ § Can be found around line 623…

Modifying unistd. h /* midfile */ #define __NR_perf_event_open 241 __SYSCALL(__NR_perf_event_open, sys_perf_event_open) #define __NR_test_newsyscall 242 __SYSCALL(__NR_test_newsyscall, sys_test_new_syscall) #undef __NR_syscalls #define __NR_syscalls 242 /* midfile */

Modifying unistd. h /* midfile */ #define __NR_perf_event_open 241 __SYSCALL(__NR_perf_event_open, sys_perf_event_open) #define __NR_test_newsyscall 242 __SYSCALL(__NR_test_newsyscall, sys_test_new_syscall) #undef __NR_syscalls #define __NR_syscalls 243 /* midfile */

Modifying syscalls. h asmlinkage long sys_perf_event_open( struct perf_event_attr __user *attr_uptr, pid_t pid, int cpu, int group_fd, unsigned long flags); #endif /* EOF */ 45

Modifying syscalls. h asmlinkage long sys_perf_event_open( struct perf_event_attr __user *attr_uptr, pid_t pid, int cpu, int group_fd, unsigned long flags); asmlinkage long sys_test_newsyscall(int test_int); #endif /* EOF */ 46

Modifying KERNEL_DIR/Makefile # Objects we will link into vmlinux / subdirs we need to visit init-y : = init/ drivers-y : = drivers/ sound/ firmware/ net-y : = net/ libs-y : = lib/ core-y : = usr/ endif # KBUILD_EXTMOD 47

Modifying KERNEL_DIR/Makefile # Objects we will link into vmlinux / subdirs we need to visit init-y : = init/ drivers-y : = drivers/ sound/ firmware/ net-y : = net/ libs-y : = lib/ core-y : = usr/ my_module/ endif # KBUILD_EXTMOD § Found around line 475… § Can replace “my_module” with the name of your PROJECT_DIR 48

Getting it all to work 1. 2. 3. 4. 5. Re-compile the kernel Install modules, install kernel Make new initramfs image Reboot Test with a user-space program…

Sample User-space Program #include <stdio. h> <stdlib. h> <sys/syscall. h> <linux/unistd. h> #define __SYS_TEST_ELEVATOR 337 int main() { int test=5; long ret; ret=syscall(__SYS_TEST_ELEVATOR, test); if(ret<0) perror("system call error"); else printf("Function successful, returned %in", ret); return 0; } 50

syscall() int syscall(int number, . . . ); n Performs the system call based on the system call’s number q Number can be found in the syscall_table_32. S file (our example was 337) 51

User-space Program Output n Output when my_module not loaded system call error: Function not implemented n Output when my_module loaded Function successful, returned 5 52

Kthreads Run the main logic of your module in a kthread! 53

Refresher: hello. c #include <linux/init. h> #include <linux/module. h> MODULE_LICENSE(“Dual BSD/GPL”); static int hello_init(void) { printk(KERN_ALERT “Hello, world!n”); return 0; } static void hello_exit(void) { printk(KERN_ALERT “Goodbye, sleepy world. n”); } module_init(hello_init); module_exit(hello_exit); 54

Kernel Modules n n Remember, kernel modules are very eventbased We need a way to start an independent thread of execution in response to an event q e. g. start_elevator() for project 2… 55

kthread_run(threadfn, data, namefmt, . . . ) n Creates a new thread and tells it to run q q q n threadfn – the name of the function the thread should run data – data pointer for threadfn (can be NULL if the function does not take any args) namefmt – name of the thread (displayed during “ps” command) Returns a task_struct 56

kthread_run example struct task_struct *t; t = kthread_run(run, NULL, “my_elevator"); if (IS_ERR(t)){ ret=PTR_ERR(t); } 57

kthread_stop int kthread_stop(struct task_struct * k); n n Sets kthread_should_stop for k to return true, wakes the thread, and waits for the thread to exit Returns the result of the thread function 58

kthread_stop_example ret=kthread_stop(t); if(ret != -EINTR) printk("Main logic tread stopped. n“); 59

Thread Function Example static int run(void *arg) { /* Lock here */ while(!kthread_should_stop()) { /* Do stuff */ /* Unlock here */ schedule(); /* Lock here */ } /* Unlock here */ printk("%s: kernel thread exits. n", __FUNCTION__); return 0; } 60

Thread Function Example static int run(void *arg) { /* Lock here */ while(!kthread_should_stop()) { /* Do stuff */ /* Unlock here */ schedule(); /* Lock here */ schedule() is very important here. Why? } /* Unlock here */ printk("%s: kernel thread exits. n", __FUNCTION__); return 0; } 61

Inefficient Solution n Thread will continue to run even though it has nothing to do q n Investigate the kthread interface to find ways to q q n Eats up resources Put thread to sleep Wake up thread There is more than one way to do this… 62

Debugging 63

Kernel Debugging Configurations n n n n n Timing info on printks __depreciated logic Detection of hung tasks SLUB debugging Kernel memory leak detector Mutex/lock debugging Kmemcheck Check for stack overflow Linked list debugging 64

Select Kernel Hacking 65

Enable Debugging Options 66

Debugging through procfs n n Necessary for elevator project! General process q q Identify data to monitor in your module Create a proc entry to monitor this data Run your module Query /proc/<entry> for that information at any time 67

Kernel Oops and Other Errors n Kernel errors often only appear on first tty (terminal interface) q n Why? How can I see my first tty? q On regular system – CTRL+ALT+F 1 n q CTRL+ALT+F 7 to go back to X screen On VMware – CTRL+ALT+SPACE+F 1 n CTRL+ALT+SPACE+F 7 to go back to X screen 68

Oops! 69

Reason for failure 70

Current drivers 71

Call Trace 72

Call Trace 73

Failed command 74

Defensive Programming • Infinite loops and deadlocks at the kernel level hang your machine – – • Ctrl-Alt-Del has NO effect Ctrl-C does not matter Ctrl-D does not matter You may only reboot How do you protect yourself? – – Use schedule() strategically Use preemptable versions of functions

Debugging Tools not Covered § § § LTT – Linux Tracing Framework gdb – Invoking gbd on the kernel image kgdb – A remote debugger for the kernel Magic Sys. Rq printk – Rate limiting, turning on/off

Next Time n n n Locks Linked lists Elevator algorithms 77