Nachos Instructional OS Part 3 CS 170 Tao

Nachos Instructional OS: Part 3 CS 170, Tao Yang, Fall 2012

Topics n n File system implementation. Assignment 3 n n Virtual memory File system extension 6/5/2021 2

Nachos File System n n Nachos Kernel Read/write open. File objects Two versions (check openfile. h and filesys. h): n A ``stub'' version is simply a front-end to the Linux. n A Nachos version implemented on top of a raw disk. Synch. Disk Function layers (from top to down): n File. System object n n Created by every call to open/access a file in a directory. Synch. Disk object n n Disk Open. File objects (many) n n Directory support. File control block (I-node) support Free block map Created by synch. Disk = new Synch. Disk("DISK") in system. cc The Disk object. n Created by disk = new Disk(name, . . ) in Synch. Disk: : Synch. Disk() 6/5/2021 3

File System Layers in Nachos Directory: Manage a directory of file names File. Header: i-node (File control block) Free. Map Bitmap for 32 x 32 blocks Nachos kernel thead File. System: Directory management file create/open/delete Open. File: access individual file with read/write/seek. Operate on bytes. Synch. Disk: synchronous access to a disk with concurrent threads. Operate on sectors. Disk: low-level interface that initiates asynchronous IO to physical disk sectors Base Operating System (Linux for our class)

Operations of File System Object n File. System(bool format). n called once at ``boot'' time. n For the stub version, just use Unix file system n Otherwise create a directory/control info. n n n Sector 0 is fileheader (i-node) for Free. Map. Sector 1 is i-node for directory table. Create(char *name, int initial. Size). n Create a Nachos disk file. ``initial. Size'' is ignored in the stub version; Unix doesn't require it. Open. File *Open(char *name). n Assume both read and write access. n in the stub version, Open simply opens the Unix file. Remove(char *name) List() 6/5/2021 5

Directory Table stored in Directory. File n n Stored in Sector 1 File content is 10 table entries. Directory. File disk space is allocated in File. System() during initialization … Bool in. Use --- is this entry used? Int sector --- location of file header (i-node) Filename -- 9 characters+ 1 for ‘n’ … 6/5/2021 6

memory directory. cc n n File. Name. Max. Len 9 // 9 characters at most for a file name Num. Dir. Entries 10 // at most 10 files Directory Block Directory Object attributes n n n Disk Limit n n Directory table. Size -- # of directory entries which is 10 now. table = in-memory directory entries, each of which is for one Nachos file. Content is copied from and copies back to a disk Directory. File. Operations n n n Directory (int size) -- Create an in-memory directory of the desired size. Fetch. From (Open. File *file) -- Fetch disk Directory. File that contains directory entries, and copy to in-memory directory table. Write. Back(Open. File *file) -- Write back in-memory directory content to disk Directory. File n n called every time when a file is added or removed to directory. Find. Index (Char *name) --Search a file string name in a directory Add (Char *name, int sector) – Add file name into a directory with disk sector no that contains the file header (i-node). Remove(char *name) – remove a file. 6/5/2021 7

File header (File control block, i-node) … int num. Bytes; // file size int num. Sectors; //sectors used int data. Sectors[Num. Direct] Data sector pointer 0 Data sector pointer 1 … Data sector pointer 29 n n Fit one sector (128 bytes) Operations n n Data sector memory File control block Allocate(Bit. Map *bit. Map, int file. Size); Fetch. From (int Sector. No) -- Fetch Fileheader sector from disk. Write. Back(int Sector. No) -- Write back in-memory file header content to disk Byte. To. Sector(int offset) –Convert a file offset in bytes to a sector no 6/5/2021 Disk File header 8

Nachos Kernel Read/write Open. File Object n n n Read(char *buffer, int num. Bytes) Write(char *buffer, int num. Bytes) Read. At(char *buffer, int num. Bytes, int File. Position). n n Synch. Disk Memory File control block Read/write in a specific position n Disk Open. File Obj Identify specific sectors in the file and fetch them, copy to buffer. Write. At(char *buffer, int num. Bytes, int File. Position) n n open. File objects Open. File(int sector) n sector – disk location of file control block. n Load FCB from disk create an in-memory Open. File object Seek(int position) Read/write in the current position May read partially written sectors first, and then write out updated sectors. Length(). Return the file size. 6/5/2021 9

Synch. Disk Object: Synchronously access a disk n n open. File objects Synch. Disk () n Called only once to create a physical disk object. Set the disk interrput hander as Requst. Done(). Synch. Disk n Create lock/semaphore for synchronization. n Notice that physical disk is an asynchronous device (disk requests return immediately, and an interrupt happens later on). Read. Sector(int sector. Number, char* buffer) n n n Send a read request for a sector Wait IO complete (wait for interrupt using a semaphone) Write. Sector(int sector. Number, char* data) n n Nachos Kernel Read/write Send a sector write request. Wait for interrupt. Request. Done() n Disk interrupt handler. Wake up any thread waiting for the disk request to finish. 6/5/2021 10

Disk Object Nachos Kernel Read/write open. File objects n n Simulates the behavior of a disk I/O device. n a single platter, with multiple tracks (32). Synch. Disk n Each track contains 32 sectors n Each sector is 128 bytes n Disk size = 32 x 32*128 =128 K Disk n Asynchronous read/write with an option of disk cache Supported operations: n Disk(char *name, Void. Function. Ptr call. When. Done, int call. Arg) n Create a disk: just create a Unix file to simulate. n Setup a handling function (Request. Done in Synch. Disk. cc) when a read/write request is done. 6/5/2021 11

Operations in Disk device Nachos Kernel Read/write open. File objects n n n Read. Request(int sector. Number, char *buffer) Synch. Disk n Read from the specific location of the “physical disk”. n ticks = Compute. Latency(sector. Number, FALSE); n interrupt->Schedule(Disk. Done, (int) this, ticks, Disk. Int); Write. Request(int sector. Number, char *data) Disk n Write data to specific location of the “physical disk”. n ticks = Compute. Latency(sector. Number, TRUE); n interrupt->Schedule(Disk. Done, (int) this, ticks, Disk. Int); Compute. Latency(int new. Sector, bool writing) n Latency= seek time + rotation time 6/5/2021 12

Assignment 3 Workload n vm directory for part 1 is empty n n n Need to come up a design and code Test C programs ~cs 170/nachosprojtest/proj 3 Amount of work n Part 1. ~350 lines of code in vm ( few changes in userprog) n n Part 2. ~200 in filesys (few in userprog) n n Sample code: ~30 lines removed from one function. Sample code: ~ 30 lines removed from one function Part 1 and 2 can be done in parallel n Part 2 may use Project 2 code without VM. 6/5/2021 13

Project 3 part 1: Virtual Memory n n Work on vm subdirectory mainly n And addrspace. cc/. h and exception. cc in userprog Create/manage a backing store (a file called SWAP using the Open. File class). Implement a page fault handler with dirty bit handling and a page replacement policy (LRU or second chance) Test under various conditions: n One process with an address space larger than physical memory. n Concurrent processes with combined address space larger than physical memory. 6/5/2021 14

Project 3 part 2: File system n Work on filesys subdirectoy mainly. n n And change system call handling code in userprog if needed (to use this extended file system). Extend the size of each file from 4 K to 100 K n A few single indirect pointers in i-node (file control block). 6/5/2021 15

Report to be submitted n 1. 2. HW 3_WRITEUP Summarize what is completed, what is not. describe the design of VM and file systems n n 3. Describe design options and their trade-offs. List/explain main components and code modification in implementing your design Summarize the test effort (what is passed, the purpose of each test) 6/5/2021 16

Emphasis of Project 3 n Play tradeoffs n n Understand the design options Deadline-driven approach n n Choose something simpler and easier to do first. Still not enough time? n Complete with minimal efforts to pass the tests. n n n Hidden tests are very similar to public test cases. No complex test cases. You are good as long as your code can run virtual memory>physical, can support files up to 100 K. Interpret the requirement towards your advantage if vague. If not specified, do something easy. 6/5/2021 17

Start with Sample Code n n Makefiles/compileable code included Part I: n n Low-level support structure is included (SWAP manager, VM page manager) Need ~30 lines n n n Implement LRU or second-chance code. Swap-out/swap-in Part II: n n n 2 -level indirect indexing is implemented Allocate 0 disk space for each file. Need ~30 lines to extend the size of each file gradually as the file expands. 6/5/2021 18

HW 3 Implementation Notes

Part 1: Getting Started n Read machine/translate. cc: n n In Machine: Translate() for virtual address translation, Page. Fault. Exception is detected when the desired page is not in memory. In Machine: Read. Mem, Translate() is called for translating the desired virtual memory address and machine->Raise. Exception() is called with Page. Fault. Exception error.

What is next n Read mipssim. cc n n Machine->Read. Mem() is called in executing each instruction. If Page. Fault. Exception is detected, the exception handler should load the desired page. The hardware will try again. Need to expand exception. cc to handle Page. Fault. Exception. n Once handled, return to user mode and restart the instruction caused the fault

User Instruction Execution Machine: Run () One. Instruction () Read. Mem () Machine: Translate() Re-execute if Exception is raised Write. Mem () Page writing? Set dirty bit Cannot find this page? Raise Page. Fault. Exception. Handler() Deal with Page. Fault. Exception

Files to be modified for Part 1 n New files in directory vm n n n Virtual memory manager Swap space manager (sample code puts at filesys) Directory userprog (extension to HW 2) n exception. cc n n Extension to handle Page. Fault. Exception Addrspace. cc/. h n n Prepare code/data for SWAP backstore. Virtual address translation -> paging if needed

Page Fault Handling n Write a Swap Manager to facilitate free sector allocation in Swap File. n n Initialize SWAPSIZE (512) as the total free sector available. Allocate a sector. Free a sector. Virtual memory manager n n Find a free memory page or a victim page Swap out a selected page to SWAP if needed load a page containing the virtual address (copy from the SWAP file to an allocated memory page). Adjust the page table.

Modify Add. Space. cc n n In translating a virtual user address for kernel, if it is not in memory, bring data from SWAP. When allocating a new address space, n n n Allocate a proper number of sectors from SWAP for this process. Copy binary data to the allocated SWAP sectors. Initial page number for this process can be 0

Synchronization issues • Two system calls may be processed concurrently and the synchronization is needed. n n n Any time a process sleeps in the kernel, another process can run Two processes try to initiate I/O on the same page at the same time May need a lock (or at least a ``busy'' flag) for each physical memory page, and possibly also for each virtual memory page.

Part 2: Steps for Nachos file system extension 1. 2. Understand how the Nachos file system operates and how Nachos implements files. Modify the file system calls from HW 2 to use Nachos file system 1. 3. Donot have to support VM (Part I/Part II can be done in parallel) Modify Nachos file system so that the file size can be upto 100 K

Files to be modified in Part 2 n Directory filesys n n n filehdr. cc/. h Support single indirect pointers in inode and can expand the file length with more sectors allocated. openfile. cc Expand file length/allocate sectors when writing data at the end or beyond. Directory userprog n exception. cc n Make sure that file system calls use extended Nachos file system.

Comparison: Unix file i-node (4 K bytes per block)

Step 3: Extend maximum file size n Modify the File. Header i-node (file control block) to support 100 K size. n Option 1: Include 5 data sector pointers, and 25 sectors for single indirect mapping. n At most map 5 + 25*32=805 blocks. Maximum file length = 805*128=103040 bytes. Option 2: single indirect mapping at every entry Suggest to add a class called Indirect. Pointer. Block which takes 1 sector (32 entries) n n n Fetch. From (int sector). Write. Back(int sector). Add. Sector (int sector) – add a new sector. Deallocate(Bit. Map *free. Map) – deallocate all sectors. Byte. To. Sector (int offset). – convert an offset in bytes to a sector.

Step 4: File size can grow n n Modify the current implementation so that a write beyond the end of the file extends the size of the file. Gracefully handle the case of the disk being full or maximum size is reached - the user process should not crash. Instead an error should be returned.

Step 4 n Modify Write. At() method in openfile. cc : n n Case 1: Write request is within the bound of the file. Handle like the existing code Case 2: Write request overwrites some portion of existing file and goes beyond by N bytes. Case 3: No overwrite to the current data, but goes beyond the end of the file. Here. To extend a file dynamically, n Extend i-node/data dynamically: include a routine in File. Header to extend a file by N bytes (by a sector or by a set of sectors)

Testing issues n n n Get sample ~cs 170/sample/PROJECT 3. Test part 1 under code/testvm using vm/nachos Test part 2. under code/testfile using filesys/nachos n Notice file names cannot exceed 9 characters.

Testing issues in Part 2 n n n Need to load user binaries/data into Nachos file system during execution Example: $ pwd cs 170/nachos/code/filesys $. /nachos -f -cp. . /test/Prog 1 -x Prog 1 Or nachos –f nachos -cp Prog 1 nachos -x Prog 1 Nachos –f Linux file Nachos –cp Prog 1 Nachos file system Nachos –x Prog 1