A comparison between xen and kvm Andrea Chierici

A comparison between xen and kvm Andrea Chierici Riccardo Veraldi INFN-CNAF CCR 2009

Outline xen n kvm n Test description n Benchmarks n Conclusions n Andrea Chierici, CCR 2009 2

What we have now: Xen Open source “industry standard” for virtualization n Included in all recent linux distributions n Supports para and hvm approaches n Widely used in HEP community n Won’t go into details during presentation n Andrea Chierici, CCR 2009 3

What could be the future: KVM n n n Kernel-based Virtual Machine Open source included in latest linux kernels: implemented as a module ¨A user space program uses /dev/kvm interface to set up VMs (qemu-kvm) n n n Supports hvm approach Rather new to HEP community Qumranet now owned by Red Hat ¨ Foreseen boost in development Andrea Chierici, CCR 2009 4

KVM virtio n Direct access to hardware ¨ Hard disk (/dev/vdx) ¨ Ethernet interface n Supported since RHEL 5. 3 ¨ kernel 2. 6. 18 -128. 1. 1 ¨ Virtio drivers must be on the VM, not on the host ¨ EGEE sw can’t currently benefit from this Andrea Chierici, CCR 2009 5

KVM seen by system admin n n rpms: basically the kernel module and a modified version of qemu (qemu-kvm) Network configuration to be done by hand in order to get public IP ¨ bridge-utils and tunctl rpms were used to set-up “tap” interface n ¨ n Script developed to automate the process A modified init. d script to configure software bridge has been developed at CNAF No VM configuration file direct support VMs are launched via a standard UNIX command + command line options ¨ libvirt ¨ Andrea Chierici, CCR 2009 6

KVM seen by system admin n No direct interface to manage VMs by command line (e. g. “xm”) ¨ Libvirt n n Virt-manager working Monitor interface ¨ Very n support powerful, lots of options available qemu already well documented, support available on-line Andrea Chierici, CCR 2009 7

KVM introduced at CNAF n We use quattor to configure and install grid nodes and virtual machines ¨ Profiles describe machine configuration ¨ Network boot for installation working n Keeps booting from network, need to restart with “-boot c” ¨ No need for any modification, just like xenhvm Andrea Chierici, CCR 2009 8

Qualitative test n n CNAF is running LHCb tier 2 site entirely on xen VMs (2 CEs, 1 SE) Changed one Computing Element with a KVM machine Has been working for more that 3 weeks flawlessly ¨ Quattor machine profile unmodified, no effort for sysadmin ¨ n n n CMS secondary squid server installed on the same host with same result (2 weeks ago) KVM executed (and live migrated!) a win 7 VM Hardware used: 1 node, dual E 5420, 16 GB ram, sata disks via Areca controller Andrea Chierici, CCR 2009 9

Quantitative test: description n n Need some measures to understand what is the best solution Tested 3 classic parameters ¨ CPU → hep-spec 06 (v 1. 1), PI test for cpu distribution ¨ Network → iperf (v 2. 0. 4) ¨ Disk access → bonnie++ (v 1. 94) n Compared Xen (para-virtualized and hvm) with KVM, using non virtualized machine as a baseline Andrea Chierici, CCR 2009 10

Quantitative test: description n Hardware used: 1 blade, dual E 5420, 16 GB ram, 10 k sas disk via LSI logic raid controller (raid 0) Xen-para VM specs: 1 vcpu, 2 GB ram, disk on a file Xen-hvm VM specs: 1 vcpu, 2 GB ram, disk on a file, “netfront” network driver n KVM VM specs: 1 vcpu, 2 GB ram, disk on a file, e 1000 network driver emulation (non-virtio tests) n n n Host OS: SL 5. 2 x 86_64, kernel 2. 6. 18 -92. 1. 22. el 5 VM OS: SLC 4. 5 i 386, kernel 2. 6. 9 -67. 0. 15. EL. cern VM OS: SLC 5. 3 x 86_64, kernel 2. 6. 18 -128. 1. 1 for kvm virtio tests KVM version: 83 Xen version: 3. 2. 1 Andrea Chierici, CCR 2009 11

Benchmarks: HEP-Spec 06 XEN vs. KVM on dual Intel E 5420, single performance measure 11 10. 5 HEP-Spec 06 10 9. 5 9 8. 5 8 7. 5 1 4 Xen-hvm Number of concurrent VMs Xen-para KVM Andrea Chierici, CCR 2009 7 8 E 5420 12

Benchmarks: HEP-Spec 06 VMs vs. CPU 70 60 HEP-Spec 06 50 40 30 20 10 0 1 vm 4 vm kvm xen-hvm 7 vm xen-para 8 vm E 5420 Andrea Chierici, CCR 2009 13

Benchmarks: HEP-Spec 06 8 VMs aggregate vs. CPUs 70 60 HEP-Spec 06 50 40 30 20 10 0 E 5410 E 5420 kvm E 5420 xen-hvm Andrea Chierici, CCR 2009 E 5420 xen-para E 5430 14

Benchmarks: HEP-Spec 06 Virtualization Technology % loss from non emulated CPU (E 5420, 8 vm) E 5420 kvm 3, 42 E 5420 xen-hvm 4, 55 E 5420 xen-para 2, 02 E 5410 vs. E 5420 4, 07 Andrea Chierici, CCR 2009 15

Benchmarks: PI test n n System tools to measure CPU usage are not reliable enough Calculating PI with n arbitrary ciphers Algorithm based on Numerical Recipes ¨ Written in C ¨ Compiled with options ¨ n n gcc -O 3 -ffast-math -finline-limit=1000 Returns the number of CPU cycles ¨ An external function included into the code calculates the number of CPU cycles Andrea Chierici, CCR 2009 16

Benchmarks: PI test KVM host vs. guest 31000000 CPU cycles 29000000 27000000 difference < 2% 25000000 23000000 21000000 host guest Andrea Chierici, CCR 2009 17

Benchmarks: PI test 8 concurrent guest 31000000 CPU cycles 29000000 27000000 25000000 23000000 21000000 guest singolo guest 1 guest 2 guest 3 guest 4 guest 5 guest 6 guest 7 guest 8 • 8 core • Balanced Performance • Every guest uses one core at 100% Andrea Chierici, CCR 2009 18

Benchmarks: PI test 17 concurrent guests 7000000 6000000 4000000 3000000 2000000 7 t 1 es gu es t 1 6 5 gu t 1 es gu es t 1 4 3 gu t 1 es 2 gu t 1 es gu es 1 0 t 1 gu es t 9 es gu t 8 es gu t 7 es gu t 6 es gu t 5 es gu t 4 es gu t 3 es gu t 2 es gu es t 1 1000000 gu CPU cycles 5000000 • Good balancing on guest • Max difference 14%, average ~5% Andrea Chierici, CCR 2009 19

Benchmarks: Iperf KVM Network Performance, e 1000 900 800 Mbits/sec 700 600 500 in 400 out 300 200 100 0 1 vm 2 vm 4 vm 8 vm host iperf -w 256 k -P 5 -t 900 Andrea Chierici, CCR 2009 20

Benchmarks: Iperf KVM vs XEN, 1 vm, network performance 1000 900 800 Mbits/sec 700 600 500 in 400 out 300 200 100 0 kvm e 1000 kvm virtio xen-para xen-hvm pv iperf -w 256 k -P 5 -t 900 Andrea Chierici, CCR 2009 21

Benchmarks: bonnie++, file vs lvm 2 GB Ram, 4 GB data set, 1 vm comparison 160000 140000 120000 K/sec 100000 80000 60000 40000 20000 0 Host 1 kvm file 1 kvm lvm 1 xen-para file 1 xen-para lvm 1 xen-hvm file 1 xen-hvm lvm bonnie++ on a single VM seq out block seq out rewrite seq in block Andrea Chierici, CCR 2009 22

Benchmarks: bonnie++ 2 GB ram, 4 GB data set, single 120000 100000 80000 60000 40000 20000 0 1 kvm vm 4 kvm vm 8 kvm vm 1 xen-para seq out block 4 xen-para seq out rewrite 8 xen-para 1 xen-hvm 4 xen-hvm 8 xen-hvm seq in block Andrea Chierici, CCR 2009 23

Benchmarks: bonnie++, kvm virtio 32 GB data set, lvm partition, single 140000 120000 K/sec 100000 80000 60000 40000 20000 0 Host hdb vdb Bonnie++ on a single vm seq out block seq out rewrite seq in block Andrea Chierici, CCR 2009 24

vir lvm m seq output block vir lvm seq output rewrite Andrea Chierici, CCR 2009 vir lvm xe tio npa xe nr pa a ra lvm xe nhv xe m nhv m lvm m kv st Ho m kv xe tio npa xe nr pa a ra lvm xe nhv xe m nhv m lvm m kv kv st Ho K/sec Benchmarks: bonnie++ 2 GB ram, 4 GB data set, 8 vm aggregate 120000 100000 80000 60000 40000 20000 0 seq input block 25

Conclusions n KVM proved good stability and reliability No problems on running production machines for more than 3 weeks ¨ CPU performances are extremely good ¨ n Modern CPU virtualization technologies help! Network performances are fair with e 1000, good with virtio ¨ Disk I/O seems the most problematic aspect ¨ n Other solutions have problems too Requires sysadmins only a small effort Even if looking promising, right now xen is the most performing solution Andrea Chierici, CCR 2009 26

Future work n qemu snapshot features ¨ Backup n and cloning of nodes high-level VM managers ¨ Ovirt, enomalism, ganeti Andrea Chierici, CCR 2009 27

Bibliography and links n Xen ¨ n Xen repository KVM ¨ Kvm repository n Quattor n HEP-SPEC Andrea Chierici, CCR 2009 28