Virtualization Operating Systems 2017 Danny Hendler Amnon Meisels

Virtualization Operating Systems, 2017, Danny Hendler & Amnon Meisels 1

What is virtualization? Creating a virtual version of something o Hardware, operating system, application, network, memory, storage “The construction of an isomorphism between a guest system and a host” [Popek, Goldberg, ’ 74] Operating Systems, 2017, Danny Hendler & Amnon Meisels 2

Example: virtual disk q Partition a single hard disk to multiple virtual disks o Virtual disk has virtual tracks & sectors q Implement virtual disk by file q Map between virtual disk and real disk contents q Virtual disk write/read mapped to file write/read in host system 3

What is virtualization? (continued) q A way to run multiple operating systems and applications on the same hardware (virtual machines) q Only virtual machine manager (a. k. a. hypervisor) has full system control q Virtual machines completely isolated from each other (or so we hope) Operating Systems, 2017, Danny Hendler & Amnon Meisels 4

Basic concepts q Virtual Machine (VM) q Host q Guest q Hypervisor (type ||) / Virtual Machine Monitor Operating Systems, 2017, Danny Hendler & Amnon Meisels 5

Basic concepts q Virtual Machine (VM) q Host q Guest q Hypervisor (type ||) / Virtual Machine Monitor Operating Systems, 2017, Danny Hendler & Amnon Meisels 6

Basic concepts q Virtual Machine (VM) q Host q Guest q Hypervisor (type ||) / Virtual Machine Monitor Operating Systems, 2017, Danny Hendler & Amnon Meisels 7

Basic concepts q Virtual Machine (VM) q Host q Guest q Hypervisor (type ||) / Virtual Machine Monitor Operating Systems, 2017, Danny Hendler & Amnon Meisels 8

Basic concepts q Virtual Machine (VM) q Host q Guest q Hypervisor (type ||) / Virtual Machine Monitor Operating Systems, 2017, Danny Hendler & Amnon Meisels 9

Types of virtualization q Full virtualization – guest OS runs unmodified q Para-virtualization – guest OS must be aware of virtualization, source-code modifications required Hardware virtualization support may be used for both Our focus is on full virtualization Operating Systems, 2017, Danny Hendler & Amnon Meisels 10

Virtualization advantages q Cost-effectiveness – less hardware o Multiple virtual machines / operating systems / services on single physical machine (server consolidation) o Various forms of computation as a service q Isolation o Good for security o Great for reliability and recovery: If VM crashes it can be rebooted, does not affect other services (fault containment) o VM migration q Development tool o Work on multiple OS in parallel o Develop and debug OS in user mode o Origins of VMware as a tool for developers Operating Systems, 2017, Danny Hendler & Amnon Meisels 11

Virtualization vs. Multi-Processing Process 1 Process 2 ∙∙∙ OS Multi-processing VM HW (disk, NIC, …) Pr 1 Pr 2 OS 1 Virtualization Pr 1 Pr 2 OS 2 User space/ kernel separation HW interface ∙∙∙ VMM/Hypervisor Virtual HW interface Real HW interface HW (disk, NIC, …) Operating Systems, 2017, Danny Hendler & Amnon Meisels 12

Type 1 and type 2 hypervisors VMware ESX, Microsoft Hyper-V, Xen VMware Workstation, Microsoft Virtual PC, Sun Virtual. Box, QEMU, KVM Figure 7 -1. Location of type 1 and type 2 hypervisors. Operating Systems, 2017, Danny Hendler & Amnon Meisels 13

What's required of a (classic) hypervisor Hypervisor should provide the following: q Safety: have full control of virtualized resources q Fidelity: program behavior on VM should be identical to its behavior on bare hardware q Efficiency: As much as possible, run directly on hardware without hypervisor intervention q Full interpretation isn't efficient Operating Systems, 2017, Danny Hendler & Amnon Meisels 15

Classic virtualization: trap and emulate VM 1 VM 2 VMM Return to process (3) HW emulation HW Trap (1) Interrupt handler (2) Emulation is the process of implementing the functionality/interface of one system on a system having different functionality/interface 16

Trap and emulate: difficulties q Sensitive instructions: behave differently in kernel/supervisor and in user mode q I/O instructions, enable/disable interrupts, … q Privileged instructions: cause a trap if executed in user mode Theorem [Popek and Goldberg, 1974] A machine can be virtualized [using trap and emulate] if every sensitive instruction is privileged. Not supported by x 86 processors prior to 2005 In 2005, Intel/AMD introduced virtualization HW support Operating Systems, 2017, Danny Hendler & Amnon Meisels 17

What is sensitive? q CPU – registers q MMU o Page table o Segments q Interrupts q Timers q I/O devices Operating Systems, 2017, Danny Hendler & Amnon Meisels 18

X 86 virtualization problem I q The x 86 architecture (w/o virtualization extensions) cannot be virtualized by trap and emulate q Some sensitive instructions are not privileged q Example: the POPF instruction o Pops 16 bits from stack to the flags register o One of the flags masks (i. e. disables) interrupts o The instruction is not privileged o What happens if the OS of a VM runs POPF ? Operating Systems, 2017, Danny Hendler & Amnon Meisels 19

X 86 virtualization problem II q Some instructions: push, pop, mov can have code segment selectors (cs, ds, ss) as arguments even in user mode, so they can be read q The selectors have two bits that are their current privilege level o x 86 (beginning with 386) has four privilege levels (ring 0 to ring 3) o The two lower bits of the cs register are the Current Privilege Level (CPL) of the code o Guest OS thinks that it is in ring 0 o Guest OS is actually in a higher ring q Result - guest OS confusion Operating Systems, 2017, Danny Hendler & Amnon Meisels 20

Implementation options q Emulation – o Full emulation – hypervisor executes code of VM step by step, testing each instruction – prohibitive overhead o Trap and emulate if sensitive instructions privileged instructions q Change sensitive instructions o Interpretation – equivalent to emulation (BOCHS, JSLinux) o Binary translation – change code on the fly (VMware, QEMU) q Para-virtualization – re-compile guest OS (XEN, Denali) q Hardware assistance – Intel VT-x and AMD-V (used by KVM, XEN, Vmware) Operating Systems, 2017, Danny Hendler & Amnon Meisels 21

Outline q Concepts, classical CPU virtualization o Binary interpretation q Memory virtualization 22

Binary translation q Binary translation is the process of translating one instruction set to another one q Approach I: translate statically all code base o In our case the result is para-virtualization o Problems § Dynamically linked libraries are not known at compile time § Self-modifying code, e. g. program generating code and running it, is not covered Operating Systems, 2017, Danny Hendler & Amnon Meisels 23

Dynamic binary translation q Approach II: translate code on the fly (Just In Time) q Simplest approach o o Keep table mapping old instructions to new instructions Fetch old instruction Use table to translate Execute new instruction(s) q Problem: performance o Overhead for every instruction similarly to interpretation Operating Systems, 2017, Danny Hendler & Amnon Meisels 24

Dynamic BT with caching q Cache translated code region: o After translation run from cache o Translation occurs only once q Static translation cannot handle dynamic control transfer, when: o Jump depending on memory address o Indirect function call (by function pointer) q Translation of dynamic control transfer must be done at execution time Operating Systems, 2017, Danny Hendler & Amnon Meisels 25

Virtualization prior to HW support Figure 7 -4. Binary translation rewrites the guest operating system running in ring 1, while the hypervisor runs in ring 0 26

VMWare binary translation: example C code Invo logg king is. P ing a r ll co ime(49), de tr ansla ted 64 -bit binary Binary (hex) representation Operating Systems, 2017, Danny Hendler & Amnon Meisels 27

VMWare binary translation: example q Translator reads guest memory at the address indicated by guest PC q Decodes instructions, creates Intermediate Representation - IR objects q Accumulates IR objects to translation units (TUs) o Basic blocks (BB), stops upon control flow First TU Compiled code fragment (CCF) Operating Systems, 2017, Danny Hendler & Amnon Meisels 28

VMWare binary translation: example q Translator reads guest memory at the address indicated by guest PC q Decodes instructions, creates Intermediate Representation - IR objects q Accumulates IR objects to translation units (TUs) o Basic blocks (BB), stops upon control flow First TU Identical code Compiled code fragment (CCF) Operating Systems, 2017, Danny Hendler & Amnon Meisels 29

VMWare binary translation: example q Translator reads guest memory at the address indicated by guest PC q Parses instructions, creates Intermediate Representation - IR objects q Accumulates IR objects to translation units (TUs) o Basic blocks (BB), stops upon control flow Translation of jump BB First TU Compiled code fragment (CCF) Operating Systems, 2017, Danny Hendler & Amnon Meisels 30

VMWare binary translation: example q Translator reads guest memory at the address indicated by guest PC q Parses instructions, creates Intermediate Representation - IR objects q Accumulates IR objects to translation units (TUs) o Basic blocks (BB), stops upon control flow Translation of fall through BB First TU Compiled code fragment (CCF) Operating Systems, 2017, Danny Hendler & Amnon Meisels 31

VMWare binary translation: example C code 64 -bit binary Invo logg king is. P ing a r ll co ime(49), de tr ansla ted Which basic block will be translated next? Operating Systems, 2017, Danny Hendler & Amnon Meisels 32

VMWare binary translation: example C code 64 -bit binary Invo logg king is. P ing a r ll co ime(49), de tr ansla ted Which basic block will be translated next? Operating Systems, 2017, Danny Hendler & Amnon Meisels 33

VMWare binary translation: example C code 64 -bit binary Invo logg king is. P ing a r ll co ime(49), de tr ansla ted 34 Operating Systems, 2017, Danny Hendler & Amnon Meisels

VMWare binary translation example: output 35

VMWare binary translation example: output These continuations remain because respective basic blocks were not executed 36

VMWare binary translation operation q Translation cache (TC) stores translations done so far q A hash table tracks the input to output correspondence q Chaining optimization allows one CCF to jump directly to another without calling out of the translation cache q As TC gradually captures guest's working set, proportion of translation decreases User code does not have to be translated Operating Systems, 2017, Danny Hendler & Amnon Meisels 37

Outline q Concepts, classical CPU virtualization o Binary interpretation q Memory virtualization 39

Memory allocation q Each VM usually receives a contiguous set of physical addresses. o 512 Mbyte – 4 Gbyte are typical values q As far as VM is concerned, this is the physical memory of the machine q The guest OS allocates pages or segments to guest processes 40

Memory management q Assumptions of OS in VM: o Physical memory is a contiguous block of addresses from 0 to some n o OS can map any virtual page to any page frame q Hypervisor must: o Partition memory among VMs o Ensure virtual page mapping only to assigned page frames q TLB – page fault in HW-managed TLB (e. g. x 86) causes HW to select a page from page table q VM OS must not manage real page table 41

Option 1: brute force Define these pages as not R/W Guest OS Page dir. CR 3 Hypervisor Page table VMM SW TLB VM memory layout Interrupt & VMM corrects address. CPU HW 42

Brute force – description q Guest page tables are read and write protected in host system. q If guest OS reads page table (e. g. for page eviction) writes page table (e. g. after page fault), or changes CR 3, the system traps. q The hypervisor then uses a VM memory layout to: q Return answers to VM q Update the layout q Hypervisor switches VM memory layout when new VM is scheduled. 43

Option 2: shadow page tables Guest OS Page dir. Hypervisor Page table VMM SW G-CR 3 TLB CPU Shadow page table Interrupt & VMM corrects page table. HW 44

Shadow page tables – description q Hypervisor maintains “shadow page tables”. q Guest page tables map: Guest VA Guest PA q Shadow tables: Guest VA Host PA. q Hypervisor does not trap guest updates to its page table. o Result – inconsistent guest page table and shadow page table. q When guest process accesses virtual address o The physical address is not in the guest page table, but in the shadow page table. o HW translates correctly, because it is aware only of shadow tables. 45

Shadow page tables – description (continued) q If address in TLB – TLB hit and no problem. q When guest process causes a page fault o Hypervisor begins execution. o Hypervisor updates guest page table with new page. o Hypervisor updates shadow page table. q Performance is as good as native execution as long as there are no page faults. q Shadow page tables should be cached so that once a VM is re-scheduled the page table does not have to be rebuilt from scratch. 46

Shadow page tables – page faults (continued) Two scenarios when handling a page fault. Hypervisor ``walks’’ guest page table to determine which it is. 1. Guest page fault – No translation in guest page tables ``inject’’ page fault for guest to handle 2. Guest translation found update shadow table respectively Operating Systems, 2017, Danny Hendler & Amnon Meisels 47

Shadow page tables – updating CR 3 Virtual CR 3 Guest Page Table Shadow Real CR 3 Page Table Slide taken from a presentation by VMWare. Page Table 48

Shadow page tables – updating CR 3 Virtual CR 3 Guest Page Table Shadow Real CR 3 Page Table Slide taken from a presentation by VMWare. Page Table 49

Shadow page tables – updating CR 3 Virtual CR 3 Guest Page Table Shadow Real CR 3 Page Table Slide taken from a presentation by VMWare. Page Table 50

Undiscovered guest page table Virtual CR 3 Guest Page Table Shadow Guest Page Table Real CR 3 Page Table Slide taken from a presentation by VMWare. Page Table 51

Undiscovered guest page table Virtual CR 3 Guest Page Table Shadow Real CR 3 Page Table Slide taken from a presentation by VMWare. Page Table 52

Option 3: nested page tables Guest OS Page dir. CR 3 Hypervisor Page table VMM SW TLB Host page table CPU EPTP HW 53

Nested page tables - description q The name implies having page tables within page tables. q The essence of the idea is a hardware assist. o Hardware has an extra pointer and the ability to walk an extra set of page tables. o Idea is called Extended Page Tables (EPT) by Intel q Guest page tables hold Guest VA Guest PA mapping, access by standard CR 3 q Extended page tables hold Host VA Host PA mapping, access by EPTP (EPT pointer). q Host VA=Guest PA 54

Walking Nested page tables 55

Nested page tables – description (cont'd) q TLB as usual holds Guest VA Host PA q On memory access o If found in TLB – no problem. o If not in TLB, but no page fault, hardware walks both tables and updates TLB. o If page fault, then hardware hypervisor gets host physical page and provides host virtual page (guest physical) to VM. 56

Sources q “Modern operating systems”, 4‘th edition, A. Tanenbaum and H. Bos q “Virtual machines”, J. E. Smith and R. Nair q A presentation by Niv Gilboa from CSE@BGU q “Formal requirements for virtualizable third generation architectures”, G. J. Popek and R. P. Goldberg, CACM, 1974 q “A comparison of software and hardware techniques for x 86 virtualization”, K. Adams and O. Ageson, ASPLOS 2006 q A presentation by VMWare Operating Systems, 2017, Danny Hendler & Amnon Meisels 57