Hyper Spector Virtual Distributed Monitoring Environments for Secure

Hyper. Spector: Virtual Distributed Monitoring Environments for Secure Intrusion Detection Kenichi Kourai Shigeru Chiba Tokyo Institute of Technology

Distributed intrusion detection system (DIDS) q Useful to achieve self-monitoring of distributed systems ◆ q analyzer Towards self-protection Consists of multiple IDSes ◆ IDS Including • Host-based IDS (HIDS) • Network-based IDS (NIDS) ◆ IDSes cooperate with each other or with an analyzer server distributed system

Threats against the DIDS q Active attacks ◆ active attack Directly take actions against IDSes by • Sending malicious packets to network • q ports used by IDSes modifying IDS policy files or terminating IDS processes Passive attacks ◆ Wait until IDSes read data including malicious code by • Sending malicious packets to monitored • servers Changing attributes of monitored files IDS monitor server passive attack

Traditional approach: Isolated monitoring DIDS q Isolates NIDSes from servers physically ◆ q Using NIDS hosts and a back-end switch NIDS hosts monitor packets by port mirroring ◆ ◆ NIDS hosts are connected to mirroring ports in a front-end switch The front-end switch duplicates and forwards packets back-end switch NIDS host server host front-end switch Internet mirroring port

Security of isolated monitoring DIDS q Prevents active attacks ◆ q The attacker cannot attack NIDS hosts using mirroring ports • Mirroring ports are only for monitoring Confines the impact of passive attacks to within the DIDS ◆ ◆ The attacker cannot access the outside of the DIDS Important because preventing passive attacks is difficult back-end switch NIDS host server host front-end switch Internet mirroring port

Problems in isolated monitoring q Need additional hardware ◆ Lots of machines for NIDSes ◆ A back-end switch ◆ A front-end switch with port mirroring q Support only NIDSes ◆ Legacy HIDSes do not support monitoring of remote server hosts ◆ Achieving secure monitoring of remote server hosts from HIDS hosts is difficult

Our approach: Hyper. Spector q Virtual ◆ IDS distributed monitoring environment • Isolate each other without • ◆A DIDS VM and server VM additional hardware The IDS VM can monitor the server VM virtual network • Connects the IDS VMs • Isolated from a network used by servers virtual network IDS VM server VM

Inter-VM monitoring mechanisms q Requirements ◆ Interfaces to legacy IDSes ◆ Secure monitoring between VMs q Hyper. Spector ◆ Software provides three mechanisms port mirroring (for packet capturing) ◆ Inter-VM disk mounting (for file system checking) ◆ Inter-VM process mapping (for process checking)

Software port mirroring q Virtual switch ◆ Achieves server VM port mirroring by NIDS software ◆ Connects its mirroring port to the IDS VM • Using a virtual network BPF device � VNI interface (VNI) and forwards packets to the IDS VM virtual switch ◆ Duplicates VMM outside mirroring port

Inter-VM disk mounting q Inter-VM ◆ Mounts disk mounter the file system of the server VM on the IDS VM • As a shadow file system ◆ Forwards requests to a shadow file system to the server VM • Using VMM interfaces server VM IDS VM file system HIDS read shadow file system � � inter-VM disk mounter VMM interface

Inter-VM process mapping q Inter-VM process mapper ◆ Maps the processes in the server VM to the IDS VM • As shadow processes ◆ Forwards • Requests to shadow processes • to the server VM Notifications from the server VM to HIDSes – Using VMM interfaces server VM IDS VM server process shadow process HIDS ptrace � � wakeup inter-VM process mapper VMM interface

Security of Hyper. Spector q Prevents active attacks ◆ ◆ q From the server VMs From hosts outside the DIDS Confines the impact of passive attacks ◆ ◆ The IDS VM cannot attack the server VM The IDS VM cannot attack hosts outside the DIDS virtual network IDS VM server VM

Security of the inter-VM monitoring mechanisms q outside hosts Secure, because ◆ ◆ The server VM cannot use inter-VM monitoring mechanisms The IDS VM cannot interfere with the server VM • Inter-VM monitoring mechanisms are only for monitoring ◆ server VM � request The IDS VM cannot send monitored information outside the DIDS • Although it can view secret information of servers. . . VMM IDS VM � monitor modify

Implementation We have implemented Hyper. Spector in the Free. BSD kernel base q IDS VM and server VM server q ◆ Based on our portspace • The portspace virtualizes only a network system, file system, and processes ◆ system VM fs net Secure enough • We assume the kernel and the base system are not exploitable VMM kernel fs IDS VM net fs

Implementation of the VMM q Implemented ◆ Virtual efficiently in the kernel switch • Maps a network interface of the server VM to the IDS VM in a read-only manner ◆ Inter-VM disk mounter • Mounts the file system of the server VM on the IDS VM read-only, using the modified union file system ◆ Inter-VM process mapper • Makes the IDS VM share the processes of the server VM in a read-only manner

Experiments q We measured overhead of Hyper. Spector ◆ Experimental setup • Snort, Tripwire, or truss in the IDS VM • thttpd in the server VM • Apache. Bench in the client host ◆ Hardware • 2 PCs (3. 0 GHz Pentium 4, • 1 GB of memory, Intel Pro/100+) 100 Base-T network switch server host IDS VM server VM client host

Snort q Monitors packets from Apache. Bench to thttpd ◆ ◆ We measured the throughput of thttpd For comparison • The base system • Isolated monitoring q Maximum overhead ◆ ◆ 7. 5% slower than the base system 7% slower than isolated monitoring (over 2 KB file size) • 30% in 0 KB file size

Tripwire q Checks the integrity of the whole file system ◆ ◆ 54, 885 objects We measured the time of the integrity check • altering the file change rate ◆ For comparison • The base system q Overhead ◆ 17 to 26% slower than the base system

Truss q Traces system calls issued by thttpd ◆ We measured the throughput of thttpd • Using Apache. Bench ◆ For comparison • The base system q Overhead ◆ 0. 8 to 7. 3% slower than the base system

Related work q Re. Virt [Dunlap’ 02], Livewire [Garfinkel’ 03] ◆ Enable IDSes to monitor servers running in a VM • The VM protects IDSes from active attacks via servers ◆ q Do not consider other attacks against IDSes Backdoors [Bohra’ 04] ◆ Enables isolated monitoring for HIDSes • Using programmable NICs to monitor server state ◆ ◆ Needs much hardware Insecure because HIDS hosts are network-reachable These need to develop specialized IDSes

Conclusion q We proposed Hyper. Spector, which ◆ Isolates IDSes from servers without additional hardware • Using IDS VMs, server VMs, and a virtual network ◆ Provides secure Inter-VM monitoring mechanisms: • Software port mirroring, inter-VM disk mounting, and inter-VM process mapping ◆ Prevents active attacks and confines the impact of passive attacks to within the DIDS

Future work q Support for active monitoring ◆ Needs a mechanism to securely send probe messages to servers q Support for Do. S attacks ◆ Needs to allocate sufficient resources to the IDS VM even under overload q Automatic detection of compromised Hyper. Spector ◆ Monitoring resource usage may help