Memory Resource Management in VMware ESX Server Carl

Memory Resource Management in VMware ESX Server Carl A. Waldspurger WMware, Inc. Palo Alto, CA carl@vwmare. com Presented by Christopher Vendome

Motivation

Motivation

Motivation

Motivation • • Inexpensive shared-memory multiprocessors Many computing environments underutilize individual servers Able to consolidate servers with little performance penalty Simplify management and reduced costs

VMware ESX Server • • • Thin software layer to multiplex hardware resources to virtual machines Manages system hardware directly o better I/O performance o complete control over resource management Novel approach to managing memory more efficiently in virtual machines

Memory Virtualization • • • Special data structure: pmap o translate physical page numbers to machine page numbers Shadow page tables maintained by processor o kept consistent with pmap o less overhead Server can remap physical pages by changing PPN-to. MPN mapping (pmap) transparent to the virtual machine

Memory Virtualization Process Physical Pages Machine Pages

Memory Virtualization Process Shadow Page Table Physical Pages Machine Pages

Reclamation Mechanisms • • • Supports overcommitment Each VM has a max size giving the illusion of a fixed physical memory o size remains constant VM will only have max size allocated when server is not overcommitted

Page Replacement Issues • • • Server needs to reclaim memory when overcommitted from VM(s) Introduce another level of paging o move physical memory pages to a swap area on disk o need to choose which VM to revoke memory and which pages in particular o meta-level policies can introduce performance anomalies Double paging problem

Ballooning • • • VM with reclaimed memory needs to operate as if it was configured with less memory Pseudo-device driver or kernel service "Inflates" when memory is reclaimed "Deflates" when memory has been freed Guest OS decides the pages to be reclaimed o native memory management is invoked not server Limitations: o uninstalled, explicitly disabled, or unavailable driver o reclaim speed o upper bounds on balloon size imposed by guest OS

Ballooning

Ballooning Evaluation

Demand Paging • • Memory is reclaimed by paging out to ESX Server swap area on disk o ESX Server coordinates asynchronous page outs Randomized page replacement policy o expectation that paging will be uncommon

Sharing Memory • • Share memory across virtual machine o same guest OS, applications, common data Less memory consumption than individual physical machines o higher levels of overcommitment

Transparent Page Sharing • • Eliminate redundant copies of pages Multiple physical pages mapped to one machine page o marked copy-on-write Requires guest OS modifications to identify redundancy Sharing can require use of restricted interfaces • Need a better solution

Content-Based Page Sharing • • • Identify page copies by contents No dependency on guest OS Hashing used to identify possible copies o successful matches followed by comparison Utilizes copy-on-write Unshared pages tagged with "hint" entry o rehashed to check for modification Varied scanning policies

Content-Based Page Sharing

Implementation • • • Hash function generates 64 -bit value o false matches ineligible for sharing Random scans of guest pages o configured to have little CPU overhead System always attempts to share a page

Page Sharing Performance

Real-World Page Sharing

Shares vs. Working Sets • • • Need quality-of-service guarantee Varied resources amongst VMs ESX Server allows administrators to control importance o maintain memory guarantees

Share-Based Allocation • • Resource rights prioritized by shares o "clients" own shares o guarantees a minimum resource fraction Dynamic min-funding revocation o revoke client with fewest shares per allocated page

Reclaiming Idle Memory • • • Proportional-share algorithms partition memory to maintain ratios o ignore active memory usage or working sets ESX Server has "idle memory tax" o reclaim idle pages Tax rate determines amount of reclaimed idle pages o extends min-funding revocation o default set to 75%

Measuring Idle Memory • • • Statistical sampling to get aggregate VM working sets o each VM is sampled separately o tracks randomly selected physical pages o samples 100 pages for 30 seconds Smoothed across samplings o exponentially-weighted averages Maximum of averages to estimate active memory

Experimental Results

Allocation Policies • • Three main parameters to control memory allocation o min, max, and share size Admission control policy ensures sufficient memory exists for the VM o min + overhead machine memory available o max - min disk swap available

Dynamic Reallocation • • Influences of memory allocation changes: o changes to system or VM allocation parameters o addition or removal of VM(s) o changes to free memory o periodic rebalancing based on idle memory ESX Server uses four reclamation states o high - 6% o soft - 4% o hard - 2% o low - 1%

Dynamic Reallocation

Dynamic Reallocation

I/O Page Remapping • • • Copying "high" memory involves a bounce buffer to push data through temporary "low" memory o more overhead, decreased performance o virtualization can make this problem worse ESX tracks frequent "high" memory pages used in I/O o remap high demand pages Current area of research o simple random replacement o adaptive approaches

Conclusions • • • Ballooning invokes guest OS memory management Idle memory tax for share-based management Content-based transparent page sharing Page remapping leveraged to reduce I/O Dynamic reallocation policy for overcommitment

Questions?