Programming Project 4 Kernel Message Passing System CS3013

Programming Project #4 Kernel Message Passing System CS-3013 Operating Systems, A-term 2009 Project Due Monday, November 23, 2009 CS-502 (EMC) Fall 2009 Kernel Message System 1

Project • To build and test a message-passing system for Inter-process Communication among separate address spaces – Implement API for message-passing functions – Kernel system calls to handle messages – Test program to exercise the message passing system CS-502 (EMC) Fall 2009 Kernel Message System 2

Objective • To gain experience programming and testing synchronization and IPC operations • To gain experience with synchronization & memory management in the Linux Kernel CS-502 (EMC) Fall 2009 Kernel Message System 3

Overview • Add a mailbox to each process – Abstract object capable of holding messages – Messages are of bounded length, undefined structure – Mailboxes may be of bounded size • All threads of a process share a mailbox • Any Linux task can send a message to any mailbox – Including own mailbox – Addressed by pid_t CS-502 (EMC) Fall 2009 Kernel Message System 4

Overview (continued) • Task may receive message only from “own” mailbox – Any thread of task may receive! • Mailbox created during fork • Mailbox deleted during process termination • Mailbox may be “stopped” – I. e. , no more message are accepted – Optionally, mailbox may be flushed CS-502 (EMC) Fall 2009 Kernel Message System 5

This Project • Predefined API (Application Program Interface) at – mailbox. h in this directory • All students must implement same API – User space interface program – Kernel implementation – Test program CS-502 (EMC) Fall 2009 Kernel Message System 6

API int Send. Msg(pid_t dest, void *msg, int len, bool block) • • Sends a message body at *msg to process dest Length = len; not more than MAX_MSG_SIZE Blocks if mailbox full and block = TRUE Returns zero if successful, error code if not int Rcv. Msg(pid_t *sender, void *msg, int *len, bool block) • • Gets a message from own mailbox, puts in *msg Sender process ID returned in *sender Blocks if mailbox empty and block = TRUE Returns zero if successful, error code if not • Messages in FIFO order CS-502 (EMC) Fall 2009 Kernel Message System 7

API (continued) int Manage. Mailbox(bool stop, int *count) • Gets number of message currently queued in mailbox • If stop = TRUE, prevents mailbox from receiving more messages – Unblocks all waiting Send. Msg and Rcv. Msg – Future Rcv. Msg calls can still retrieve remaining queued messages • Returns zero if successful, error code if not CS-502 (EMC) Fall 2009 Kernel Message System 8

API — Documented Error Codes • MAILBOX_FULL • MSG_TOO_LONG • Non-blocking send • MAILBOX_EMPTY • On send • MSG_ARG_ERROR • Non-blocking receive • Invalid argument or pointer • copy_to_user or copy_from_user fails • MAILBOX_STOPPED • On any send • Also after blocked send or receive • MAILBOX_INVALID • On any call CS-502 (EMC) Fall 2009 • MAILBOX_ERROR • Any other kind of error • You may add other error codes as needed Kernel Message System 9

Kernel Implementation • Start with pre. Patch-Project 4 • I. e. , so we all have same task_struct and header files • Three system calls. long sys_mailbox_send. long sys_mailbox_rcv. long sys_mailbox_manage • Create and delete mailboxes • Memory allocation within kernel • All messages of fixed size • Synchronization within kernel • Suggest simulating a monitor per mailbox CS-502 (EMC) Fall 2009 Kernel Message System 10

Kernel Implementation • Start with pre. Patch-Project 4 • I. e. , so we all have same task_struct • Three system calls. long sys_mailbox_send. long sys_mailbox_rcv. long sys_mailbox_manage • Create and delete mailboxes • Memory allocation within kernel • All messages of fixed size • Synchronization within kernel CS-502 (EMC) Fall 2009 Kernel Message System 11

Pre-Patch your Kernel • Apply pre. Patch-Project 4 to a clean kernel tree • Adds to task_struct • struct cs 3013_mailbox *mailbox • spinlock_t mailbox_lock • Defines syscalls in syscall. S and unistd. h. long sys_mailbox_send /* 333 */. long sys_mailbox_rcv /* 334 */. long sys_mailbox_manage /* 335 */ • Reason • Graders’ time in grading • Don’t modify other commonly used header files! CS-502 (EMC) Fall 2009 Kernel Message System 12

Kernel Implementation • Start with pre. Patch-Project 4 • I. e. , so we all have same task_struct • Three system calls. long sys_mailbox_send. long sys_mailbox_rcv. long sys_mailbox_manage • Create and delete mailboxes • Memory allocation within kernel • All messages of fixed size • Synchronization within kernel CS-502 (EMC) Fall 2009 Kernel Message System 13

Creating a Mailbox • During do_fork() • In kernel/fork. c • If this is a new process • Use kmalloc() to allocate a new data structure for a new mailbox • Initialize • Set pointer in task_struct->mailbox Initialize mailbox_lock • If this is just a new thread, keep existing mailbox • Indicated by CLONE_THREAD argument to do_fork() • Copy mailbox pointer Initialize mailbox_lock CS-502 (EMC) Fall 2009 Kernel Message System 14

Creating a Mailbox (continued) • No mailbox for kernel threads, etc. • Indicated by null task_struct->mm • Set task_struct->mailbox to null CS-502 (EMC) Fall 2009 Kernel Message System 15

Kernel Implementation of Mailbox Operations • Monitor model per mailbox – Three system calls for user-visible functions sys_mailbox_send sys_mailbox_rcv sys_mailbox_manage – Two other functions for kernel support mailbox_create mailbox_destroy • Contents of monitor data structure – Monitor lock • spinlock_t initialized to SPIN_LOCK_UNLOCKED • Mostly likely, this is different from mailbox_lock in task_struct – Semaphore and count to simulate condition variable • Both initialized to zero – List head for linked list of messages; flag for stopped mailbox – Other fields as needed CS-502 (EMC) Fall 2009 Kernel Message System 16

Kernel Implementation (continued) • Rcv. Msg • Grab monitor lock • While linked list is empty – Simulate wait on condition variable • Unlink first message from mailbox linked list • Release monitor lock • copy_to_user to copy message body and other information to caller • Free kernel space for unlinked message (see below) • Non-blocking receive • Exercise for student CS-502 (EMC) Fall 2009 Kernel Message System 17

Kernel Implementation (continued)if • Rcv. Msg • Grab monitor lock • While linked list is empty k ed c e h opp c to is st d e Ne ilbox ma – Simulate wait on condition variable • Unlink first message from mailbox linked list • Release monitor lock • copy_to_user to copy message body and other information to caller • Free kernel space for unlinked message (see below) • Non-blocking receive • Exercise for student CS-502 (EMC) Fall 2009 Kernel Message System 18

Kernel Implementation (continued) • Rcv. Msg s n o ati er p o f o r e rd ? o – Simulate wait on condition variable y te h o W N e. linked • Unlink first message from mailbox list her • Release monitor lock • copy_to_user to copy message body and other information to caller • Free kernel space for unlinked message (see below) • Grab monitor lock • While linked list is empty • Non-blocking receive • Exercise for student CS-502 (EMC) Fall 2009 Kernel Message System 19

Kernel Implementation (continued) • Rcv. Msg • Grab monitor lock • While linked list is empty re u l i a f – Simulate wait on condition variable f o d re? n i • Unlink first message from mailbox linked list e k h t a cur h • Release monitor lock W oc n a c • copy_to_user to copy message body and other information to caller • Free kernel space for unlinked message (see below) • Non-blocking receive • Exercise for student CS-502 (EMC) Fall 2009 Kernel Message System 20

Kernel Implementation (continued) • Send. Msg • Allocate kernel space for new message (see below) • copy_from_user to copy message body into kernel • Fill in other details (sender, length, etc. ) • Grab monitor lock • Link new message to end of mailbox linked list • Simulate signal to condition variable if any receivers waiting • Release monitor lock • Blocking send • Exercise for student • Must simulate wait on condition variable if mailbox is full CS-502 (EMC) Fall 2009 Kernel Message System 21

Kernel Implementation (continued) • Manage. Mailbox • • Grab monitor lock Count number of messages If stop = TRUE, determine if any tasks waiting If so, trick them into “unblocking” and returning error • Release monitor lock • Revisit blocking Send. Msg, Rcv. Msg • When unblocked, be sure mailbox has not been stopped in meantime. CS-502 (EMC) Fall 2009 Kernel Message System 22

Deleting a Mailbox • In do_exit() • Only if the “task group” is dead! • Stop the mailbox to be sure that blocked send operations can complete! – Flush messages (to free their space) dition ) n ow l o e cdone eb e – Make sure all blocked operations are e c s ( a • Free the mailbox data – kfree() r e l e t ub cur h structure s A oc can • Zero out task_struct->mailbox CS-502 (EMC) Fall 2009 Kernel Message System 23

Memory Allocation Resources • kmalloc(), kfree() • linux/slab. h • Similar to malloc() & free(), but with flags • Slab allocator • kmem_cache_t* kmem_cache_create() • void* kmem_cache_alloc() • void kmem_cache_free() • int kmem_cache_destroy() CS-502 (EMC) Fall 2009 Kernel Message System 24

Memory Allocation Resources (continued) • Use slab allocator – All message bodies the same size in kernel – Highly optimized for rapid allocation and free – Low fragmentation • Initialization – Establish cache and any static variables during fork_init() (in kernel/fork. c) – All mailboxes share same cache CS-502 (EMC) Fall 2009 Kernel Message System 25

Locking Tools in the Kernel • Mailbox • Needs to be locked to link and unlink messages • Also to change state (STOP) • Remember – the Linux Kernel is fully preemptive! • • System call may be preempted before completion Interrupt may schedule another process at any time Interrupt may manage shared resources But not while holding a spinlock – Due to support for symmetric multi-processing CS-502 (EMC) Fall 2009 Kernel Message System 26

Robert Love says … • It is a major bug if … – An interrupt occurs to access a resource while kernel code is also manipulating that resource – Kernel code is preempted while accessing a shared resource – Kernel code sleeps while in the middle of a critical section – Two processors access same data at same time • Implementing locking is not hard • Tricky part is identifying what to lock. CS-502 (EMC) Fall 2009 Kernel Message System 27

Tools for Simulating Monitors • Spin locks (for monitor locking) • spin_lock_init() • spin_lock(), spin_unlock() • See linux/spinlock. h • Must be used to provide mutual exclusion for all monitor functions • Also to release and acquire lock around condition variable waits CS-502 (EMC) Fall 2009 Kernel Message System 28

Condition Variable (simulated) • Condition variable representation • Struct semaphore, int wait_count, both initialized to zero • See asm/semaphore. h • Wait on condition variable /* while holding the monitor lock */ wait_count++; /* incr # of waiting tasks */ spin_unlock(&mr_lock); /*release lock */ down_interruptible(&sem); /* wait */ spin_lock(&mr_lock); /* reacquire lock */ • Signal condition variable /* while holding the monitor lock */ if (wait_count > 0) up(&sem); /* unblock a task */ wait_count--; /* decr # of waiting tasks */ CS-502 (EMC) Fall 2009 Kernel Message System 29

Monitor Example* monitor FIFOMessage. Queue { /* function implementations */ struct q. Item { struct q. Item *next, *prev; msg_t msg; }; FIFOMessage. Queue(void) { /* internal data of queue*/ }; struct q. Item *head, *tail; condition non. Empty; void add. Msg(msg_t new. Msg) { /* function prototypes */ void add. Msg(msg_t new. Msg); msg_t remove. Msg(void); /* constructor/destructor */ FIFOMessage. Queue(void); ~FIFOMessage. Queue(void); q. Item *new = malloc(q. Item); new prev = tail; new next = NULL; if (tail==NULL) head = new; else tail next = new; tail = new; signal non. Empty; }; * /* constructor*/ head = tail = NULL; Adapted from Kleiman, Shah, and Smaalders CS-502 (EMC) Fall 2009 Kernel Message System 30

Monitor Example /* function implementations continued*/ /* function implementations concluded*/ msg_t remove. Msg(void) { ~FIFOMessage. Queue(void) { while (head == NULL) wait(non. Empty); /* destructor*/ while (head <> NULL) { struct q. Item *old = head; if (old next == NULL) tail = NULL; /*last element*/ else old next prev = NULL; head = old next; msg_t msg = old msg; free(old); struct q. Item * top = head; head = top next; free(top); }; /* what is missing here? */ /* Answer: - need to unblock waiting threads in destructor! */ return(msg); }; CS-502 (EMC) Fall 2009 Kernel Message System 32

Deleting mailbox • Must stop mailbox first • In a loop, unblock all waiting tasks • Be sure any waiting task has had time to complete operation • Potential race condition • Flush remaining messages • Delete mailbox data structure CS-502 (EMC) Fall 2009 Kernel Message System 33

Subtle Race Condition • Process A may be sending msg to Process B • Send. Msg gets copy of pointer B->mailbox • Gets pre-empted while in kernel! • Process B tries to exit! • Deletes mailbox structure • Zeros out B->mailbox • Exits cleanly • Process A gets to top of ready queue • Is dispatched • Continues to send to B->mailbox. Big trouble! CS-502 (EMC) Fall 2009 Kernel Message System 34

Subtle Race Condition (continued) • mailbox_lock in task_struct provided for your convenience • You may use it however you wish • You may ignore it and do something else • No single “correct” solution • Suggestion: – mailbox_lock protects the mailbox pointer, not the mailbox itself • Perhaps an atomic integer is useful (p. 132) CS-502 (EMC) Fall 2009 Kernel Message System 35

Testing • A little bit like Project 3, but … • Multiple processes • Each process with multiple threads • Fork multiple processes • Create mailboxes, exchange mailbox IDs • Randomly send messages to each other • Payload must be self identifying • Acknowledge received messages • Test extreme conditions • E. g. , fill up mailbox CS-502 (EMC) Fall 2009 Kernel Message System 36

Due Dates • Project due at Monday, November 23 • Pace yourself: – – By November 2: – • Creation & deletion of mailboxes in do_fork(), do_exit() • Design of mailbox data structure for producer-consumer – By November 9: – • Sending and receiving messages • User space support and simple test program – By November 16: – • Stopping mailbox with blocked calls on Send. Msg or Rcv. Msg • Flushing messages • Comprehensive test program – By November 23: – • Solve subtle race condition CS-502 (EMC) Fall 2009 Kernel Message System 37

Final Submission • Submit using my. WPI • Include – One patch file for kernel implementation • Difference from pre. Patch-Project 4 and your implementation – User space implementation of mailbox. h interface – Test program(s) and results – Makefile – Write-up must explain synchronization & programming invariants • Sending and receiving messages • Shutting down mailboxes and race condition • Test program Put your name on all documents & at top of every edited file! CS-502 (EMC) Fall 2009 Kernel Message System 38

Individual vs. Teams • This may be an individual project or a 2 person team project • If two people wish to be a team, inform instructor by on November 2 • Even if you have worked together on previous projects CS-502 (EMC) Fall 2009 Kernel Message System 39

Questions? CS-502 (EMC) Fall 2009 Kernel Message System 40