Programming Project 2 Message Passing System CS 502

Programming Project #2 Message Passing System CS 502 Operating Systems Fall 2007 CS-502 Fall 2007 Project 2 1

Project • To build and test a message-passing system for Inter-process Communication among separate address spaces – Implement API for message-passing functions – Kernel system calls to handle messages – Test program to exercise the message passing system CS-502 Fall 2007 Project 2 2

Objective • To gain experience programming and testing synchronization and IPC operations • To gain experience with synchronization & memory management in the Linux Kernel CS-502 Fall 2007 Project 2 3

Overview • Add a mailbox to each process – Abstract object capable of holding messages – Messages are of bounded length, undefined structure – Mailboxes may be of bounded size • All threads of a process share a mailbox • Any Linux task can send a message to any mailbox – Including own mailbox – Addressed by pid_t CS-502 Fall 2007 Project 2 4

Overview (continued) • Task may receive message only from “own” mailbox – Any thread of task may receive! • Mailbox created during fork • Mailbox deleted during process termination • Mailbox may be “stopped” – I. e. , no more message are accepted – Optionally, mailbox may be flushed CS-502 Fall 2007 Project 2 5

This Project • Predefined API (Application Program Interface) at – mailbox. h in this directory • All students must implement same API – User space interface program – Kernel implementation – Test program CS-502 Fall 2007 Project 2 6

API int Send. Msg(pid_t dest, void *msg, int len, bool block) • • Sends a message body at *msg to process dest Length = len; not more than MAX_MSG_SIZE Blocks if mailbox full and block = TRUE Returns zero if successful, error code if not int Rcv. Msg(pid_t *sender, void *msg, int *len, bool block) • • Gets a message from own mailbox, puts in *msg Sender process ID returned in *sender Blocks if mailbox empty and block = TRUE Returns zero if successful, error code if not • Messages in FIFO order CS-502 Fall 2007 Project 2 7

API (continued) int Manage. Mailbox(bool stop, int *count) • Gets number of message currently queued in mailbox • If stop = TRUE, prevents mailbox from receiving more messages – Unblocks all waiting Send. Msg and Rcv. Msg – Future Rcv. Msg calls can still retrieve remaining queued messages • Returns zero if successful, error code if not CS-502 Fall 2007 Project 2 8

API — Documented Error Codes • MAILBOX_FULL • MSG_TOO_LONG • Non-blocking send • On send • MAILBOX_EMPTY • MSG_ARG_ERROR • Non-blocking receive • MAILBOX_STOPPED • On any send • Also after blocked send or receive • MAILBOX_ERROR • Any other kind of error • MAILBOX_INVALID • You may add other error codes as needed • On any call CS-502 Fall 2007 • Invalid argument or pointer • copy_to_user or copy_from_user fails Project 2 9

Kernel Implementation • Start with pre. Patch-Project 2 • I. e. , so we all have same task_struct and header files • Three system calls. long sys_mailbox_send. long sys_mailbox_rcv. long sys_mailbox_manage • Create and delete mailboxes • Memory allocation within kernel • All messages of fixed size • Synchronization within kernel • Suggest simulating a monitor per mailbox CS-502 Fall 2007 Project 2 10

Kernel Implementation • Start with pre. Patch-Project 2 • I. e. , so we all have same task_struct • Three system calls. long sys_mailbox_send. long sys_mailbox_rcv. long sys_mailbox_manage • Create and delete mailboxes • Memory allocation within kernel • All messages of fixed size • Synchronization within kernel CS-502 Fall 2007 Project 2 11

Pre-Patch your Kernel • Apply pre. Patch-Project 2 to a clean kernel tree • Adds to task_struct • struct *mailbox • Defines syscalls in syscall. S and unistd. h. long sys_mailbox_send /* 318 */. long sys_mailbox_rcv /* 319 */. long sys_mailbox_manage /* 320 */ • Reason • Instructor’s time in grading • Don’t modify other commonly used header files! CS-502 Fall 2007 Project 2 12

Kernel Implementation • Start with pre. Patch-Project 2 • I. e. , so we all have same task_struct • Three system calls. long sys_mailbox_send. long sys_mailbox_rcv. long sys_mailbox_manage • Create and delete mailboxes • Memory allocation within kernel • All messages of fixed size • Synchronization within kernel CS-502 Fall 2007 Project 2 13

Creating a Mailbox • During do_fork() • In kernel/fork. c • If this is a new process • Use kmalloc() to allocate a new data structure for a new mailbox • Initialize • Set pointer in task_struct->mailbox • If this is just a new thread • Keep existing mailbox of process • Indicated by CLONE_THREAD argument to do_fork() CS-502 Fall 2007 Project 2 14

Creating a Mailbox (continued) • No mailbox for kernel threads, etc. • Be sure task_struct->mailbox is null CS-502 Fall 2007 Project 2 15

Kernel Implementation of Mailbox Operations • Monitor model per mailbox – Three system calls for user-visible functions sys_mailbox_send sys_mailbox_rcv sys_mailbox_manage – Two other functions for kernel support mailbox_create mailbox_destroy • Contents of monitor data structure – Monitor lock • spinlock_t initialized SPIN_LOCK_UNLOCKED – Semaphore and count to simulate condition variable • Both initialized to zero – List head for linked list of messages; flag for stopped mailbox – Other fields as needed CS-502 Fall 2007 Project 2 16

Kernel Implementation (continued) • Rcv. Msg • Grab monitor lock • While linked list is empty – Simulate wait on condition variable • Unlink first message from mailbox linked list • Release monitor lock • copy_to_user to copy message body and other information to caller • Free kernel space for unlinked message (see below) • Non-blocking receive • Exercise for student CS-502 Fall 2007 Project 2 17

Kernel Implementation (continued) • Rcv. Msg • Grab monitor lock • While linked list is empty if k c ped e h c top o t s d s i e Ne ilbox ma – Simulate wait on condition variable • Unlink first message from mailbox linked list • Release monitor lock • copy_to_user to copy message body and other information to caller • Free kernel space for unlinked message (see below) • Non-blocking receive • Exercise for student CS-502 Fall 2007 Project 2 18

Kernel Implementation (continued) • Rcv. Msg • Grab monitor lock • While linked list is empty f o r e ere d r o h s e t n – Simulate wait on condition variable No ratio e p o • Unlink first message from mailbox linked list • Release monitor lock • copy_to_user to copy message body and other information to caller • Free kernel space for unlinked message (see below) • Non-blocking receive • Exercise for student CS-502 Fall 2007 Project 2 19

Kernel Implementation (continued) e r u • Rcv. Msg il a f f • Grab monitor lock o re? nd he • While linked list is empty i k ur t – Simulate wait on condition variable ha occ W list n • Unlink first message from mailbox linked a c • Release monitor lock • copy_to_user to copy message body and other information to caller • Free kernel space for unlinked message (see below) • Non-blocking receive • Exercise for student CS-502 Fall 2007 Project 2 20

Kernel Implementation (continued) • Send. Msg • Allocate kernel space for new message (see below) • copy_from_user to copy message body into kernel • Fill in other details (sender, length, etc. ) • Grab monitor lock • Link new message to end of mailbox linked list • Simulate signal to condition variable if any receivers waiting • Release monitor lock • Blocking send • Exercise for student • Must simulate wait on condition variable if mailbox is full CS-502 Fall 2007 Project 2 21

Kernel Implementation (continued) • Manage. Mailbox • • Grab monitor lock Counter number of messages If stop = TRUE, determine if any tasks waiting If so, trick them into “unblocking” and returning error • Release monitor lock • Revisit blocking Send. Msg, Rcv. Msg • When unblocked, be sure mailbox has not been stopped in meantime. CS-502 Fall 2007 Project 2 22

Deleting a Mailbox • In do_exit() • Only if the task group is dead! • Stop the mailbox to be sure that blocked send operations can complete! – Flush messages (to free their space) • Free the mailbox data structure – kfree() • Zero out task_struct->mailbox CS-502 Fall 2007 Project 2 23

Memory Allocation Resources • kmalloc(), kfree() • linux/slab. h • Similar to malloc() & free(), but with flags • Also vmalloc() – slight performance hit but does not require physically contiguous pages • Slab allocator • kmem_cache_t* kmem_cache_create() • void* kmem_cache_alloc() • void kmem_cache_free() • int kmem_cache_destroy() CS-502 Fall 2007 Project 2 24

Memory Allocation Resources (continued) • Use slab allocator – All message bodies the same size in kernel – Highly optimized for rapid allocation and free – Low fragmentation • Initialization – Establish cache and any static variables during init. c CS-502 Fall 2007 Project 2 25

Locking Tools in the Kernel • Mailbox • Needs to be locked to link and unlink messages • Also to change state (START, STOP) • Remember – the Linux Kernel is fully preemptive! • • System call may be preempted before completion Interrupt may schedule another process at any time Interrupt may manage shared resources But not while holding a spinlock – Due to support for symmetric multi-processing CS-502 Fall 2007 Project 2 26

Robert Love says … • It is a major bug if … – An interrupt occurs to access a resource while kernel code is also manipulating that resource – Kernel code is preempted while accessing a shared resource – Kernel code sleeps while in the middle of a critical section – Two processors access same data at same time • Implementing locking is not hard • Tricky part is identifying what to lock. CS-502 Fall 2007 Project 2 27

Tools for Simulating Monitors • Spin locks (for monitor locking) • spin_lock_init() • spin_lock(), spin_unlock() • See linux/spinlock. h • Must be used to provide mutual exclusion for all monitor functions • Also to release and acquire lock around condition variable waits CS-502 Fall 2007 Project 2 28

Condition Variable (simulated) • Condition variable representation • Struct semaphore, int wait_count, both initialized to zero • See asm/semaphore. h • Wait on condition variable /* while holding the monitor lock */ wait_count++; /* incr # of waiting tasks */ spin_unlock(&mr_lock); /*release lock */ down_interruptible(&sem); /* wait */ spin_lock(&mr_lock); /* reacquire lock */ • Signal condition variable /* while holding the monitor lock */ if (wait_count > 0) up(&sem); /* unblock a task */ wait_count--; /* decr # of waiting tasks */ CS-502 Fall 2007 Project 2 29

Monitor Example* monitor FIFOMessage. Queue { /* function implementations */ struct q. Item { struct q. Item *next, *prev; msg_t msg; }; FIFOMessage. Queue(void) { /* internal data of queue*/ }; struct q. Item *head, *tail; condition non. Empty; void add. Msg(msg_t new. Msg) { /* constructor*/ head = tail = NULL; q. Item *new = malloc(q. Item); new prev = tail; new next = NULL; /* function prototypes */ void add. Msg(msg_t new. Msg); msg_t remove. Msg(void); if (tail==NULL) head = new; else tail next = new; /* constructor/destructor */ tail = new; FIFOMessage. Queue(void); ~FIFOMessage. Queue(void); signal non. Empty; }; * Adapted from Kleiman, Shah, and Smaalders CS-502 Fall 2007 Project 2 30

Monitor Example /* function implementations continued*/ /* function implementations concluded*/ msg_t remove. Msg(void) { ~FIFOMessage. Queue(void) { while (head == NULL) wait(non. Empty); /* destructor*/ while (head <> NULL) { struct q. Item *old = head; if (old next == NULL) tail = NULL; /*last element*/ else old next prev = NULL; head = old next; struct q. Item * top = head; head = top next; free(top); }; /* what is missing here? */ msg_t msg = old msg; free(old); return(msg); }; CS-502 Fall 2007 Project 2 31

Monitor Example /* function implementations continued*/ /* function implementations concluded*/ msg_t remove. Msg(void) { ~FIFOMessage. Queue(void) { while (head == NULL) wait(non. Empty); /* destructor*/ while (head <> NULL) { struct q. Item *old = head; if (old next == NULL) tail = NULL; /*last element*/ else old next prev = NULL; head = old next; struct q. Item * top = head; head = top next; free(top); }; /* what is missing here? */ /* Answer: - need to unblock waiting threads in destructor! */ msg_t msg = old msg; free(old); return(msg); }; CS-502 Fall 2007 Project 2 32

Deleting mailbox • Must stop mailbox first • In loop, unblock all waiting tasks • Be sure any waiting task has had time to complete operation • Potential race condition • Flush remaining messages • Delete mailbox data structure CS-502 Fall 2007 Project 2 33

Testing • Fork multiple processes • Create mailboxes, exchange mailbox IDs • Randomly send messages to each other • Payload must be self identifying • Acknowledge received messages • Test extreme conditions • E. g. , fill up mailbox CS-502 Fall 2007 Project 2 34

Due Dates • Project due at start of class, Monday, October 22 • Pace yourself: – – October 8: – • Creation & deletion of mailboxes in do_fork(), do_exit() • Design of mailbox data structure for producer-consumer – October 15: – • Sending and receiving messages • User space support and test program – October 22: – • Stopping mailbox with blocked calls on Send. Msg or Rcv. Msg • Flushing messages CS-502 Fall 2007 Project 2 35

Submission • Submit using web-based turnin program – http: //turnin. cs. wpi. edu: 8088/servlets/turnin. ss • Include – One patch file for kernel implementation • Difference from pre. Patch-Project 2 and your implementation – User space implementation of mailbox. h interface – Test program(s) and results – Makefile – Write up explaining implementation and testing • Put your name on all documents and at top of every edited file! CS-502 Fall 2007 Project 2 36

Questions? CS-502 Fall 2007 Project 2 37