Kthreads Mutexes and Debugging Sarah Diesburg CS 3430

Kthreads, Mutexes, and Debugging Sarah Diesburg CS 3430 Operating Systems 1

Refresher: procfs Kernel Module n procfs “hello world” example q n Creates a read/write procfs entry Steps q Create entry in module_init function n q Registers reading/writing function using file_operations structure Delete entry in module_cleanup function 2

Testing Procfs #> #> #> insmod hello_proc. ko tail /var/log/syslog cat /proc/helloworld echo “hi” > /proc/helloworld rmmod hello_proc 3

Project 3: Remember n You will write your own proc module called remember that q q Allows the user to write a string to /proc/remember (max length 80) Allows the user to read /proc/remember and get back the string that was just added n If no string was added, the string EMPTY should be read instead 4

Remember Example #> echo “Hello there” > /proc/remember #> cat /proc/remember Hello there #> 5

Kthreads Run the main logic of your module in a kthread! 6

Refresher: hello. c #include <linux/init. h> #include <linux/module. h> MODULE_LICENSE(“Dual BSD/GPL”); static int hello_init(void) { printk(KERN_ALERT “Hello, world!n”); return 0; } static void hello_exit(void) { printk(KERN_ALERT “Goodbye, sleepy world. n”); } module_init(hello_init); module_exit(hello_exit); 7

Kernel Modules n n Remember, kernel modules are very eventbased How can we start an independent thread of execution to run in the background? 8

kthread_run(threadfn, data, namefmt, . . . ) n Creates a new thread and tells it to run q q q n threadfn – the name of the function the thread should run data – data pointer for threadfn (can be NULL if the function does not take any args) namefmt – name of the thread (displayed during “ps” command) Returns a task_struct 9

kthread_run example struct task_struct *t; t = kthread_run(run, NULL, “my_thread"); 10

kthread_stop int kthread_stop(struct task_struct * k); n n Sets kthread_should_stop for k to return true, wakes the thread, and waits for the thread to exit Returns the result of the thread function 11

kthread_stop_example ret=kthread_stop(t); if(ret != -EINTR) printk("Main logic tread stopped. n“); 12

Thread Function Example static int run(void *arg) { /* Lock here */ while(!kthread_should_stop()) { /* Do stuff */ } /* Unlock here */ printk("%s: kernel thread exits. n", __FUNCTION__); return 0; } 13

Demo kthread module 14

Part 3: Kitchen Scheduling 15

Part 3: Kitchen Scheduling n n Implement a /proc kernel module that simulates an busy kitchen at a restaurant Your /proc/kitchen file must accept writes of different dishes q n Each dish takes some time to process Your /proc/kitchen file must return status information when read 16

Why Scheduling? n n Classic producer/consumer analogy Similar to disk schedulers q q File system produces read/write requests Disk consumes requests, optimized for disk head position, rotational delays, etc. 17

Your Kitchen n n One kitchen 20 order slots q n Use a static array of ints? Four types of dishes q q 1 - Caesar Salad 2 - Hamburger 3 - Personal Pizza 4 - Beef Wellington 18

Food orders n A write of 1 -4 to /proc/kitchen will fill an order slot with the dish corresponding to the number q n You decide how the order slots get filled (e. g. round robin or other way) Kitchen takes 1 second to look in an order slot q Regardless if empty or containing an order 19

Food Orders n n The kitchen can only process one order at a time Each order takes a different amount of time to process q q 2 seconds for a Caesar Salad 3 seconds for a Hamburger 4 seconds for a Personal Pizza 8 seconds for a Beef Wellington 20

Kernel Time Constraints #include <linux/delay. h> void ssleep(unsigned int seconds); n A call to ssleep will have the program cease to the task scheduler for seconds number of seconds 21

Food Order n Once an order is processed, the kitchen can mark that order slot as empty and should look at other order slots to find more orders to process. 22

Processing the orders? n n As soon as the kitchen module loads, it should start a persistent background thread that looks at each slot in the array and processes orders The actual order processing logic will be run in a kthread q This is where your module should ssleep to look in the queue and “process” an order 23

Status Information n Performing a read on /proc/kitchen should return some status information q q q n Current spot being looked at in the queue Current order being processed Number of orders processed If the kitchen is not running, the last two pieces of information do not need to be displayed 24

Additional Design Considerations n n n How to place orders in order slots? How to check order slots from the kitchen? What scheduling algorithm to use? 25

Demo n n n Project 3 Specification Working binary Kitchen starter code 26

Concurrency Aspects of Project 3 n Synchronizing access to request queue q q n Multiple producers may access request queue(s) at the same time Multiple consumers may access request queue(s) at the same time Synchronizing access to other global data

Kitchen Queue Concurrency Jobs may appear on the queue at the same time the kitchen module checks the queue The status may be read at the same time that you're updating n n q q q n Number of jobs that you've serviced Which job is currently being processed Which slot in the queue kitchen is looking at How do you guarantee correctness?

Global Data vs. Local Data n Global data is declared at global scope, e. g. outside of any function body q n Often necessary for kernel programming Particularly sensitive to concurrency issues q Be extra careful when handling globals

Global Data vs. Local Data n n Local data is declared within a function Local data is sensitive to concurrency issues when it depends on global data or when parallel access is possible q Think carefully about whether it needs to be synchronized 30

Synchronization Primitives n Semaphores q q n Mutexes q q n User space Kernel space Spin locks q n User space Kernel space Atomic Functions

Synchronization Primitives (We'll Only Cover These) n Mutexes q q n User space Kernel space Does anyone remember the differences between a mutex and semaphore?

The Mutex Caught up in the mutex? 33

Mutexes n n Mutex – A construct to provide MUTual EXclusion Based on the concept of a semaphore Found at <source_dir>/include/linux/mutex. h Can be locked or unlocked q Only one thread of execution may hold the lock at a time

Kernel-Space Mutex Initialization n mutex_init(&mutex) n Declare and initialize a mutex q Only initialize it once

Kernel-Space Mutex - Locking n void mutex_lock(struct mutex *); n int n n mutex_lock() can wait indefinitely mutex_lock_interruptible() locks a mutex as long as it is not interrupted q n mutex_lock_interruptible(struct mutex *); returns 0 if locking succeeded, < 0 if interrupted Use of the interruptible version is typically preferred

Kernel-Space Mutex – Unlocking n void mutex_unlock(struct mutex *); n Guarantees that the mutex is unlocked q Why is there no interruptible version of this function? 37

Kernel-Space Mutex Example /* Declare your mutex */ struct mutex my_mutex; /* Initialize your mutex */ mutex_init(&my_mutex); /* Lock */ if(mutex_lock_interruptible(&my_mutex)) return -ERESTARTSYS; /* Do stuff to protected global variables */ /* Unlock */ mutex_unlock(&my_mutex); 38

User-Space Mutex n Also used with pthreads in regular user applications q q pthreads operate very similar to kthreads Might be useful if you are prototyping your code in user-space before porting to kernel 39

User-Space Mutex Initialization n int pthread_mutex_init(pthread_mutex_t *, NULL); n int pthread_mutex_destroy(pthread_mutex_t *); n n Pthread_mutex_init() dynamically allocates a mutex Pthread_mutex_destroy() frees a mutex

User-Space Mutex - Locking n int pthread_mutex_lock(pthread_mutex_t *); n Returns 0 on locking, < 0 otherwise

User-Space Mutex - Unlocking n int pthread_mutex_unlock(pthread_mutex_t *); n Returns 0 on unlocking, < 0 otherwise

Kitchen Scheduling Advice 43

General Advice n Just make kitche work first q q n Use a very simple algorithm My kitchen searches the queue in round-robin fashion and processes every maintenance request Optimize if there is time 44

Round Robin n Method: q n Service requests in round-robin fashion (e. g. queue slot 0, 1, 2, 3, etc. ) Pros/Cons?

Shortest Job First n Method: q n Service fastest orders first Pros/Cons?

Hybrid n n Combine methods, come up with something new Up to your creativity 47

Getting Help n n n Class time to answer questions Regular office hours Email 48

Other Hints n This is not a simple project q q q Setup is different You need to use different files and methods of compilation/running Do NOT wait until 2 days before it is due to start n q Too late The Internet will likely NOT be very helpful 49

Other Hints n n Set milestones Ask questions early q If it’s a good question, I’ll share it with the class 50

Debugging 51

Kernel Debugging Configurations n n n Timing info on printks Detection of hung tasks Kernel memory leak detector Mutex/lock debugging Kmemcheck Check for stack overflow 52

Select Kernel Hacking 53

Enable Debugging Options 54

Debugging through reads to /proc/penguin n n Necessary to help you “see” what’s going on! General process q q q Identify data to monitor in your module Run your module Query (read from) /proc/penguin for that information at any time 55

Kernel Oops and Other Errors n Kernel errors often only appear on first tty (terminal interface) q Why? 56

Oops! 57

Reason for failure 58

Current drivers 59

Call Trace 60

Call Trace 61

Failed command 62