Malware Detection via Virtual Machine Monitoring Wenke Lee

Malware Detection via Virtual Machine Monitoring Wenke Lee

Host-Based Monitoring • Monitor host activities to detect the presence of malware – Compromised applications, malware, etc. • Run malware on host (a “controlled” environment”, e. g. , a sandbox) to understand its behavior – Analyze malware runtime data to understand malicious activities, input conditions, etc. • The host monitors and malware analyzers need to be tamper resistant and transparent to malware

Host-Based Monitoring (cont’d) • Virtual machine provides the separation required for tamper resistance and transparency – Security VM monitors the application VMs – Challenges: timely and fine-grained • Current research projects – Memory monitoring framework – Malware analysis tools/platform

Why Monitor Memory? • The only reliable source on the current state of a computer system is memory • Nearly endless data for security, forensics, etc – Running and (some) killed processes – Encryption keys and decrypted data – Network sockets and data – OS-level accounting information – User input (e. g. , key strokes, mouse movement) – Screen captures and graphical elements – And much more!

Monitoring Memory on Production-Level Systems (1) Passive Monitoring: Viewing memory in Ai from S without any timing synchronization between the two virtual machines (2) Active Monitoring: Viewing memory in Ai from S with event notification being sent from Ai to S to permit monitoring at relevant times (3) Locating Valuable Data: Applying models obtained from supervised learning to find critical data structures within the raw memory view

Passive Monitoring application periodically views memory from another virtual machine -- technique known as VM introspection Mapping “raw memory view” to virtual addresses and symbols requires the steps shown in figure below. Address and symbol mapping can be performed by a VM introspection library (e. g. , Xen. Access) BD Payne, M Carbone, and W Lee. Secure and Flexible Monitoring of Virtual Machines. In ACSAC 2007.

The Xen. Access Library • The only open source VM introspection library • Access to virtual addresses, kernel symbols, and more • Works with Xen and dd-style memory image files • Released in Spring 2006 • Maintained by GTISC to encourage more research • http: //www. xenaccess. org

Active Monitoring application receives event notification from Guest VM when code execution reaches one of the hooks installed in the Guest VM kernel. Hooks and all associated code are protected from tampering using hypervisor-enforced memory protections (i. e. , User VM can not modify these security-critical components). Hooks invoke trampoline, which transfers control to the security application. BD Payne, M Carbone, M Sharif, and W Lee. An Arch for Secure Active Monitoring Using Virtualization. In Oakland 2008.

Ether • Use Intel VT hardware virtualization extensions to provide instruction execution on actual hardware • Extend the Xen hypervisor to leverage Intel VT for malware analysis • Provides for both instruction-by-instruction examination of malware, and also coarser grained system call-by-system call examination • System Diagram:

Ether: Experiments • We created two tools to test the Ether framework: – Ether. Unpack: extracts hidden code from obfuscatd malware – Ether. Trace: Records system calls executed by obfuscated malware • We then compared both of these tools to current academic and industry approaches – Ether. Unpack: we compared how well current tools extract hidden code by obfuscating a test binary and looking for a known string in the extracted code – Ether. Trace: we obfuscated a test binary which executes a set of known operations, and then observe if they were logged by the tool

Ether: Ether. Unpack Results Packing Tool Poly. Unpack Renovo Ether. Unpack Armadillo no no yes Aspack no yes Asprotect yes yes FSG yes yes MEW yes yes Mole. Box no yes Morphine yes yes Obsidium no no yes PECompact no yes yes Themida VM no no yes UPX yes yes UPX Scrambled yes yes Win. UPack no yes Yoda’s Protector no yes

Ether: Ether. Trace Results Packing Tool Norman Sandbox Anubis Ether. Trace None yes yes Armadillo no no yes UPX yes yes Upack yes yes Themida yes yes PECompact yes yes ASPack yes yes FSG yes yes ASProtect yes no yes Win. Upack yes yes t. Elock yes no yes PKLITE 32 yes yes Yoda’s Protector no yes Ns. Pack yes yes MEW yes yes n. Pack yes yes RLPack yes yes RCryptor yes yes

Thank You!