Efficient VM Introspection in KVM and Performance Comparison

Efficient VM Introspection in KVM and Performance Comparison with Xen Kenichi Kourai Kousuke Nakamura Kyushu Institute of Technology

Intrusion Detection System (IDS) ¡IDSes detect attacks against servers ¡Monitor the systems and networks of servers ¡Alert to administrators ¡Recently, attackers attempt to disable IDSes ¡Before they are detected ¡This is easy because IDSes are running in servers detect IDS server intrude

IDS Offloading ¡Offloading IDSes using virtual machines (VMs) ¡Run a server in a VM ¡Execute IDSes outside the VM ¡Prevent IDSes from being compromised ¡Can be provided as a cloud service ¡Cloud providers can protect users' VMs VM monitor IDS In-VM monitoring IDS VM IDS offloading

VM Introspection (VMI) ¡A technique for monitoring VMs from the outside ¡Memory introspection ¡Obtain raw memory contents and extract OS data ¡Disk introspection ¡Obtain raw disk data and interpret a filesystem ¡Network introspection VM ¡Obtain packets only from/to VMs IDS packets ? ? ? memory ? ? ? disk network

Performance of VMI ¡Performance has not been reported in detail ¡No performance comparison ¡E. g. , VMwatcher [Jiang+ CCS'07] ¡Implemented in Xen, QEMU, VMware, and UML ¡Reported only for UML ¡E. g. , EXTERIOR [Fu+ VEE'13] ¡Implemented in KVM and QEMU ¡No difference due to using memory dump ¡Performance data is important ¡For user's selection of virtualization software

The Purpose of This Work ¡Performance comparison among virtualization software in terms of VMI ¡Target: Xen and KVM ¡Widely used open source virtualization software ¡System architecture is different process VM VM VM hypervisor OS Xen KVM

Implementation for KVM ¡No efficient implementation of VMI for KVM ¡Several studies have been done for KVM ¡The implementation details are unclear ¡Lib. VMI KVM [Payne+ '11] supports VMI for both Xen and ¡The performance of memory introspection is too low in KVM ¡Optimized for Xen

KVMonitor ¡We have developed an efficient VMI tool for KVM ¡Execute an IDS as a process of the host OS ¡Provide functions for introspecting memory, disks, and NICs in QEMU VM offload IDS KVMonitor disk monitor host OS NIC memory KVM module QEMU

Memory Introspection (1/2) ¡Difficult to efficiently introspect QEMU's memory ¡Lib. VMI obtains memory contents from QEMU ¡KVMonitor shares VM's physical memory with QEMU via a memory file ¡Access As a memory-mapped file ¡Enable direct memory introspection IDS VM KVMonitor QEMU VM's physical memory file

Memory Introspection (2/2) ¡IDSes usually access OS data using virtual addresses ¡KVMonitor translates virtual addresses into physical addresses ¡Look up the page table for address translation ¡Introspect the CR 3 register using QMP IDS page table CR 3 VM KVMonitor QEMU VM's physical memory file

Disk/Network Introspection ¡KVMonitor introspects VM's disks via the network block device (NBD) ¡Interpret the qcow 2 format in the NBD server ¡Interpret the filesystem in the host OS ¡KVMonitor captures packets from a tap device IDS KVMonitor NBD disk image file VM NBD server host OS QEMU tap network

Transcall with KVMonitor ¡We have ported Transcall KVM [Iida+ '11] for Xen to ¡Enable offloading legacy IDSes without any modifications ¡Consist of a system call emulator and a shadow filesystem ¡Including the proc filesystem ¡Analyze OS data by memory introspection IDS Transcall KVMonitor VM analyze QEMU

Experiments ¡We examined that KVMonitor achieved ¡Efficient memory introspection ¡No impact on memory performance of a VM ¡Effective IDS offloading PC VM CPU: Intel Xeon E 5630 (12 MB L 3 cache) Memory: 6 GB DDR 3 PC 3 -8500 HDD: 250 GB SATA NIC: gigabit Ethernet Hypervisor: KVM 1. 1. 2 Host OS: Linux 3. 2. 0 CPU: 1 Memory: 512 MB Disk: 20 GB (ext 3) Guest OS: Linux 2. 6. 27

KVMonitor vs. Lib. VMI ¡We measured the performance of memory introspection KVMonitor Lib. VMI ¡KVMonitor was ¡ 32 x faster than Lib. VMI read (GB/s) ¡Copy VM's physical memory by 4 KB 12 10 9. 6 fast 8 6 4 2 0 0. 3

Why is Lib. VMI so slow? ¡Lib. VMI has to issue a QMP command for each memory access ¡Memory contents are transferred from QEMU to Lib. VMI IDS QMP Lib. VMI VM IDS QEMU KVMonitor VM's memory VM memory file KVMonitor QEMU VM's memory

In-VM Memory Performance ¡Doesn't using a memory file affect memory performance of a VM? memory file ¡as efficient as malloc memory file VM VM QEMU VM's memory file malloc throughput (GB/s) ¡Using a memory file was 10 9 8 7 6 5 4 3 2 1 0 malloc 8. 6 8. 5 6. 6 6. 3 read write

KVMonitor vs. In-VM Access ¡KVMonitor was faster than in-VM memory access KVMonitor ¡Due to virtualization overhead 10 VM IDS KVMonitor VM's memory file QEMU VM's memory read (GB/s) 8 6 4 2 0 In-VM 9. 6 8. 6 fast

Offloading Legacy IDSes (1/3) ¡Tripwire ¡Check filesystem integrity in disks ¡We added, deleted, and modified files ¡Offloaded Tripwire detected changed files Rule Name. . . Added Removed Modified Monitor Filesystems 1 1 1 Total Objects scanned: 67082 Total violations found: 3 VM Tripwire DB disk

Offloading Legacy IDSes (2/3) ¡Snort ¡Inspect network packets ¡We performed portscans from another host ¡Offloaded Snort detected portscans [**] [1: 1421: 11] SNMP Agent. X/tcp request [**] [Classification: Attempted Information Leak]. . . 01/28 -10: 47: 13. 406931 192. 168. 0. 68: 47962 -> 192. 168. 0. 81: 705 Snort rule sets packets VM portscan

Offloading Legacy IDSes (3/3) ¡Chkrootkit ¡Detect rootkits using ps, netstat, and file inspection ¡We tampered with ps and netstat in a VM ¡Offloaded chkrootkit detected tampered commands ROOTDOR is ’/’ Checking ’ps’. . . INFECTED Checking ’netstat’. . . INFECTED : execute ps VM disk chkrootkit netstat . . . ps netstat

Cross-view Diff (1/2) ¡A technique for detecting hidden malware ¡Compare the results of VMI and in-VM monitoring ¡The difference means the existence of hidden malware C is hidden cross-view diff engine A B C D. . . VM monitor IDS A B D. . . IDS

Cross-view Diff (2/2) ¡We tampered with ps in a VM ¡A hidden process was detected as malicious ¡We tampered with netstat in a VM ¡A hidden port was detected as a backdoor ps netstat PID TTY TIME CMD 1 ? 00: 00 init 2 ? 00: 00 kthreadd : PID TTY TIME CMD 2 ? 00: 00 kthreadd : Proto. . . Local Address. . . tcp 0. 0: 5900 tcp 0. 0: 22 : Proto. . . Local Address. . . tcp 0. 0: 22 : results from offloaded commands results from in-VM commands

KVMonitor vs. Xen ¡We compared the performance of VMI between KVM and Xen ¡Using a VMI tool for Xen ¡Memory: standard library ¡Disk: loopback mount ¡Network: tap device Dom 0 (VM) disk image file tap IDS Hypervisor: Xen 4. 1. 3 Dom 0 OS: Linux 3. 2. 0 VM: fully virtualized VM libxenctrl hypervisor

Memory Introspection ¡We measured read throughput ¡Copy VM's physical memory by 4 KB 12 ¡ 48 x faster than Xen read (GB/s) ¡KVMonitor was 10 KVM Xen 9. 6 fast 8 6 4 2 0 VMI

Why is Xen so slow? ¡Xen has to map each memory page ¡It cannot map all the pages in advance ¡It takes time proportional to the number of pages ¡KVMonitor can read a pre-mapped file VM IDS libxenctrl KVMonitor map Xen VM's memory file KVMonitor

Kernel Integrity Checking ¡We measured the execution time of the kernel integrity checker ¡Read the code area ¡Translate virtual to physical addresses ¡ 118 x faster than Xen 250 Xen 224 200 time (ms) ¡KVMonitor was KVM 150 100 fast 50 0 1. 9

Why is the speedup so larger? ¡The speedup in the real IDS was much larger ¡ 48 x (simple benchmark) ¡ 118 x (kernel checker) ¡Due to address translation ¡In Xen, the access cost of the page table is high ¡Only 8 bytes are read after memory mapping VM IDS libxenctrl map & read simple benchmark VM IDS libxenctrl map & read real kernel checker

Disk Introspection ¡We measured the execution time of Tripwire ¡For two formats of disks KVM ¡raw and qcow 2 9. 4 9. 2 10 ¡Comparable to Xen 8 time (min) ¡KVMonitor was ¡The difference between formats was larger ¡Raw was faster than qcow 2 Xen 7. 5 6 4 fast 2 0 raw qcow 2

Network Introspection ¡We measured the packet loss rate in Snort ¡Send many packets as fast as possible ¡more lightweight than Xen ¡Dom 0 suffered from virtualization overhead 12 packet loss rate (%) ¡KVMonitor was KVM Xen 10. 4 10 8 6. 2 6 4 2 0 fast

Chkrootkit ¡We measured the execution time of chkrootkit KVM ¡KVMonitor was 60 ¡ 1. 6 x faster than Xen ¡ 2 x slower than in-VM ¡Due to system call traps 55 50 time (sec) ¡Efficient memory introspection ¡No virtualization overhead Xen 40 35 fast 30 18 20 21 10 0 Offloading in-VM

Related Work ¡VMI tools ¡Livewire [Garfinkel+ NDSS'03] for VMware ¡Xen. Access [Payne+ ACSAC'07] for Xen ¡Shm-snapshot for Lib. VMI [Xu+ PDL'13] ¡Take a VM's memory snapshot in shared memory ¡It takes 1. 4 seconds for 3 GB ¡Volatility [Walters '07] ¡A memory forensics framework ¡VMI for KVM is enabled by a Python adapter, Py. VMI from Lib. VMI

Conclusion ¡KVMonitor ¡Achieve efficient VM introspection (VMI) in KVM ¡ 32 x faster than existing Lib. VMI ¡Performance comparison with Xen ¡ 118 x faster at maximum ¡Chkrootkit was 1. 6 x faster ¡Future work ¡Comparison with other virtualization software ¡Integration with Lib. VMI