CSC 382582 Computer Security Integrity Management CSC 382582

Models of Intrusion Detection 1. Misuse detection – You know what’s bad. – Attempt

Theory of Malicious Code Theorem 22 -1: It is undecidable whether an arbitrary program

Detecting Malware Signature-based – Look for known patterns in malicious code. – Defeated by

Detecting Malware Code Emulation – Execute potential malware on VM. – Scan VM memory

Detecting Malware Heuristics – Code execution starts in last section. – Suspicious code redirection.

Detecting Malware Behavior-based – Watch for known actions from malicious code. – Network access

References 1. 2. 3. 4. 5. 6. 7. 8. 9. Ross Anderson, Security Engineering,

Slides: 8

Download presentation

CSC 382/582: Computer Security Integrity Management CSC 382/582: Computer Security 1

Models of Intrusion Detection 1. Misuse detection – You know what’s bad. – Attempt to detect bad items. – Anti-virus and anti-spyware tools. 2. Anomaly detection – You know what’s good. – Attempt to detect deviations from good state. – Host Intrusion Detection Systems (HIDS). CSC 382/582: Computer Security 2

Theory of Malicious Code Theorem 22 -1: It is undecidable whether an arbitrary program contains a computer virus. Proof: Define virus v as TM program that copies v to other parts of the tape, while not overwriting any part of v. Reduce to Halting Problem: T’ running code V’ reproduces V iff running T on V halts. Theorem 22 -2: It is undecidable whether an arbitrary program contains malicious logic. CSC 382/582: Computer Security 3

Detecting Malware Signature-based – Look for known patterns in malicious code. – Defeated by polymorphic viruses. Smart scanning – Skips junk instructions inserted by poly engines. – Skips whitespace/case changes in macro viruses. Decryption – Brute-forces simple XOR-based encryption. – Checks decrypted text against small virus sig to decide whether has plaintext or not. CSC 382/582: Computer Security 4

Detecting Malware Code Emulation – Execute potential malware on VM. – Scan VM memory after certain # iterations. – Watch instructions for decryptor profile. Code Optimization. – Optimize away junk instructions and odd techniques used by polymorphic viruses. CSC 382/582: Computer Security 5

Detecting Malware Heuristics – Code execution starts in last section. – Suspicious code redirection. – Suspicious section ACLs or size. – Suspicious library routine imports. – Hard-coded pointers into OS kernel. Neural Network Heuristics – IBM researchers trained neural net to recognize difficult polymorphic viruses. – Released in Symantec antivirus. CSC 382/582: Computer Security 6

Detecting Malware Behavior-based – Watch for known actions from malicious code. – Network access signature of worm. – Unexpected use of dangerous system calls. Integrity Checking – Host-based Intrusion Detection System. – Record MAC, size, dates, ACL of files. – Periodically check for changes. – ex: Tripwire, AIDE CSC 382/582: Computer Security 7

References 1. 2. 3. 4. 5. 6. 7. 8. 9. Ross Anderson, Security Engineering, Wiley, 2001. Matt Bishop, Computer Security: Art and Science, Addison-Wesley, 2003. William Cheswick, Steven Bellovin, and Avriel Rubin, Firewalls and Internet Security, 2/e, 2003. Fred Cohen, http: //www. all. net/books/virus/part 1. html, 1984. Simson Garfinkel, Gene Spafford, and Alan Schartz, Practical UNIX and Internet Security, 3/e, O’Reilly & Associates, 2003. Cyrus Peikari and Anton Chuvakin, Security Warrior, O’Reilly & Associates, 2003. Ed Skoudis and Lenny Zeltser, Malware: Fighting Malicious Code, Prentice Hall, 2003. Ed Skoudis, Counter Hack Reloaded 2/e, Prentice Hall, 2006. Peter Szor, The Art of Computer Virus Research and Defense, Addison. Wesley, 2005. CSC 382/582: Computer Security 8