Live Migration of Virtual Machines Christopher Clark Keir

Live Migration of Virtual Machines Christopher Clark, Keir Fraser, Steven Hand, Jacob Gorm Hanseny, Eric July, Christian Limpach, Ian Pratt, Andrew Wareld

Index • • • Motivation Migration Pre-copy migration Challenge in Pre-copy Implementation Managed Migration(Xen) Self Migration Experiment Optimization

Motivation • Why we need live migration? We want to move VM without interrupting VM service. Products: • Xen 2. 0 : live migration • VMware: VMotion

Migration • Two important consideration: 1. Downtime 2. Total migration time • All migrations mechanism can be divided into three phases. 1. Push phase 2. Stop-and-copy phase 3. Pull phase

Migration Example: • Stop-and-copy => Only stop-and-copy phase • Pure demand-migration => Stop-and-copy phase and Pull phase • Pre-copy => Push phase and Stop-and-copy phase

Pre-copy migration

Challenge in Pre-copy • Every VM have some set of pages which updates very frequently. • And the Sets size would influence downtime of the migration. ÞWritable Working Sets • The bandwidth of network also impact the migration time.

Implementation • There are two kinds of implementation of migration. – Managed Migration • The physical machine has a management VM. (Xen) – Self Migration • There are three conditions to change into stopand-copy phase. – The dirty rate is bigger than the upper bound. – The size if working set is small enough. – The limit of the rounds.

Managed Migration(Xen) 1. 1 st round: – – – Copy all memory pages to destination machine. Replace original page table with shadow page table whose all pages are marked read-only. Create a dirty bit map for the VM. 2. 2 nd-(n-1)th round: – – – During the pages transferring, if VM want to modify a page, it will invoke Xen to set the appropriate bit in the dirty bit map. Dirty pages will be resend again. Reset dirty bit map for the next round. 3. nth round: – When the Dirty rate is bigger than up bound, begin to do stopand-copy.

Self Migration 1. 2. 1 st round: – – – 2 nd-(n-1)th round: – – – 3. Copy all memory pages to destination machine. Mark all virtual address as write-protect. The OS create a dirty bit map(used to tracking physical pages) to memory the modified pages. During the pages transferring, if write faults occur, OS will set the appropriate bit in the dirty map. Dirty pages will resend again. Reset dirty bit map for the next round. nth round: – When the dirty rate is bigger than the upper bound, OS would disable all activities. 4. Copy all dirty pages to shadow buffer. 5. Transfer the buffer and ignore new page updates.

Web Server migration

SPECweb 99 benchmark

A worst case example

Optimization • Dynamic Rate-Limiting • Rapid Page Dirtying • Para-virtualized Optimization 1. Ballooning mechanism 2. Stunning rogue Process