Lesson 13 Unix Forensics Overview Basic MO email

Lesson 13 Unix Forensics

Overview • Basic MO (email sent) • System Tools • Files System Tools • Using the Tools UTSA IS 6353 Incident Response

Basic MO • Only use tools you can trust • Consider using tools stored off-line on readonly medium • Don’t forget a rogue kernel module can make a compromised system look clean • Hackers who use Unix are considered to be above average threats, innovative, and persistent UTSA IS 6353 Incident Response

Basic System Response Tools(1) • vmstat: provides quick look a memory, CPU, and disk subsystem statistics • mpstat: provides processor utilization statistics • ps: process status, shows processes executing on the system and information about them • top: like ps, but also orders the processes by CPU consumption UTSA IS 6353 Incident Response

Basic System Response Tools(2) • iostat: provides statistics on disk subsystem • sar: displays system perfomance data stored over-time • lastcom: displays commands that have been run on the system UTSA IS 6353 Incident Response

Basic File Tools • lsof: list open files, show all files the operating system has open • file: outputs a list of hex values that indicate the file contents • readelf: displays the executable linking and format headers of a binary file so you can determine what functions the executable performs UTSA IS 6353 Incident Response

Basic File Tools(2) • ldd: reads the contents of the ELF headers to show what shared object libraries the executable depends • strace: starts a process or attaches to a currently running process so you can see all the system calls the process is making(Solaris: truss, BSD: ktrace) UTSA IS 6353 Incident Response

Basic Search Tools(2) • strings: shows ASCII strings >= 4 characters or longer (user specified) in a file • find: recursively searches for objects on the file system—can be useful for finding files or directories based on user specified criteria • grep: global regular expression print—searches a pattern sapce (file) for a user specified pattern UTSA IS 6353 Incident Response

Basic Premise of the Hack • Target Machine is s Webserver running Apache • A Webmaster calls the Incident Response Team to say that the CPU is busy on the server which gets very few daily hits • Your job to prove or dispel that an incident has occured UTSA IS 6353 Incident Response

Tool Usage to Uncover Hacker • Step One: On-line Analysis or off-line analysis? On-line! • vmstat: shows a user process is running • mpstat: memory not being taxed • iostat: very little disk activity • sar: shows CPU started being used at 03: 17 • Who could use the CPU at 03: 17? • Sys. Admin? Easy to confirm UTSA IS 6353 Incident Response

Tool Usage to Uncover Hacker(2) • lastcom: shows FTP was run by root several times • last: shows recent logins, but it doesn’t show where logged in from • top –d 1: shows process Apache using CPU 100% • ps –eflcym –headers (for linux) – Shows Apache and multiple httpd processes UTSA IS 6353 Incident Response

Tool Usage to Uncover Hacker(3) • lsof – p – Shows apache process has a file john. pot open – Google search on john. pot points to John the Ripper – Good enough for me to say an incident has occurred—but why stop now? • file apache – Shows apache is an ELF executable UTSA IS 6353 Incident Response

Tool Usage to Uncover Hacker(4) • ldd. /apache – Shows fewer binary links • ldd /usr/sbin/httpd – Shows many more binary links • Looks like apache is a rogue process • String apache|grep – i john – Shows apache is probably the password cracker John the Ripper • So Why would the Webserver need John the Ripper? UTSA IS 6353 Incident Response

So What Do We Know? • The password cracker John the Ripper found on our Web server • Hack “appears” to have occurred at 03: 17 AM • But don’t be fooled by time hacks—what is server tine synced with? • Did hacker change the time? UTSA IS 6353 Incident Response

Things to Consider • Tools we will use execute in user space • If root kit changed binaries then tool results could be inaccurate – Do off-line analysis if you suspect • Best approach on “live” systems is to execute from read only media – Ensures the tools aren’t modified by hacker – Knoppix Linux is another useful tool UTSA IS 6353 Incident Response

Off-line Analysis • Unix allows you to mount volumes in readonly mode mount –o loop=/dev/loop 0 /var/tmp/testfilesystem /mnt • To find out which file systems supported find /lib/modules/’uname – r’ /kernel/fs/* - type f|grep – v ‘n/s/’ • To see what file systems are loaded grep –v ‘^nodev’ /proc/filesystems UTSA IS 6353 Incident Response

Mounting A File System Read-Only mounting a file system image in Linux: ##== mount the image read-only so that the image doesn't change on disk # mount -o ro, loop=/dev/loop 0 /var/space/images/2003_02_17_linux. bin /mnt ##== list of files # ls -la /mnt total 146 drwxr-xr-x 19 root 1024 Feb 17 2003. drwxr-xr-x 19 root 4096 Sep 12 11: 55. . -rw-r--r-1 root 0 Sep 12 11: 55. autofsck -rw------1 root 186 Sep 6 19: 58. bash_history drwxr-xr-x 2 root 2048 Feb 8 2003 bin drwxr-xr-x 3 root 1024 Sep 6 19: 59 boot drwxr-xr-x 20 root 116736 Feb 17 2003 dev drwxr-xr-x 41 root 3072 Sep 12 14: 13 etc [ output deleted ] ##== unmount /mnt # umount /mnt UTSA IS 6353 Incident Response

Tools to Detect Changes NATIVE • Linux – rpm –Vva • Solaris (9. 0) – pkgchk –vn • Debian (3. 0) • • 3 rd Party Tools AIDE Integrit Osiris Tripwire – Debsums -ac Tools run periodically and rely on hashes or checksums UTSA IS 6353 Incident Response

Tools to Detect in Real-time • • changedfiles dnotify FAM All use kernel modules or poll the file system to detect changes in near realtime UTSA IS 6353 Incident Response

MAC Times • Modify, access, change times • Unix files systems sotre a set of timestamps in their file system metadata • Timestamps stored in inode • mtime, atime, ctime UTSA IS 6353 Incident Response

How common commands change MACtimes for a directory (foo): Action atime ctime X X X creation (mkdir foo) mtime directory move (mv foo bar) X X file creation (touch foo/foo) X X file creation (dd if=/dev/zero of=foo/foo count=1) X X list directory (ls foo) change directory (cd foo) X file test (-f foo) file move/rename (mv foo_mvd) X X permissions change (chmod/chown <some_perm> foo) X file copy (mv foo_mvd foo) X X file edit (vim foo) X X file edit (emacs foo) X X X file edit (nvi/nano foo)

How common commands change MACtimes for a file (f 1): Action atime ctime mtime creation (touch foo) X X X creation (dd if=/dev/zero of=foo count=1) X X X rename (mv foo bar) permissions change (chmod <some_perm> foo) copy (cp foo bar) X X copy overwrite (cp bar foo) X X append (cat >> foo) X X overwrite (cat > foo) X X truncate (cp /dev/null foo) X X list file (ls foo) edit (vim/emacs/xemacs/joe/jed foo) X X X edit (ed/nvi/vi (sun)/vi (obsd)/nano/pico foo) X 1 X 1 1 - all times changed, but atime is slightly older than mtime and ctime

Using ls to Display MAC times displaying MACtimes using ls: Linux (ls from GNU fileutils) Open. BSD Solaris mtime ls -latr --full-time ls -lat. Tr ls -latr atime ls -laur --full-time ls -lau. Tr ls -laur ctime ls -lacr --full-time ls -lac. Tr ls -lacr UTSA IS 6353 Incident Response

So Now What? • Search filesystem for files that changed/modified around suspect time # find / (-mtime +2 –a –mtime -7) –a –printf “M: %t C: %c %it%pn” Or # find / (-mtime +2 –a –mtime -7)|perl –ne ‘chomp; ($i, $m, $c)=(stat)[1, 9, 10]; printf “M: %st$it$_n”, localtime($m). ” C: ”. localtime($c)’ • Finding broken suid or sgid binary to escalate privilege find / -perm -6000 -ls UTSA IS 6353 Incident Response

Using Find the find command in action: ##== find all files which have had their status changed in ##== the last 24 hrs (display ctime, inode, and filename) # find / -ctime -1 -printf "%c %it%pn" Fri Sep 7 11: 55: 14 2003 174945 /var/lib/rpm Fri Sep 7 11: 55: 14 2003 174946 /var/lib/rpm/__db. 001 Fri Sep 7 11: 55: 17 2003 174947 /var/lib/rpm/__db. 002 Fri Sep 7 11: 55: 17 2003 174948 /var/lib/rpm/__db. 003 Fri Sep 7 11: 55 2003 160125 /var/lib/random-seed Fri Sep 7 11: 55: 16 2003 222706 /var/log Fri Sep 7 12: 17: 05 2003 224545 /var/log/messages [ output deleted ] ##== find all files which have had their status changed from ##== 5 to 30 (inclusive) days ago and display the ctime, inode, and filename # find / -ctime +4 -ctime -31 -printf "%c %it%pn" Sat Aug 30 19: 49: 52 2003 414661 /boot/System. map-2. 4. 20 -20. 8 bigmem Sat Aug 30 19: 49: 52 2003 414662 /boot/config-2. 4. 20 -20. 8 bigmem Sat Aug 30 19: 49: 52 2003 414663 /boot/module-info-2. 4. 20 -20. 8 bigmem Sat Aug 30 19: 49: 53 2003 414664 /boot/vmlinux-2. 4. 20 -20. 8 bigmem Sat Aug 30 19: 49: 53 2003 414665 /boot/vmlinuz-2. 4. 20 -20. 8 bigmem [ output deleted ] UTSA IS 6353 Incident Response

File System Debugger • fsdb – Open. BSD and Solaris • debugfs – ext 2 and ext 3 (Linux) • Debugfs <image path> UTSA IS 6353 Incident Response

Steps Investigator Took • • • First Key Step: on-line or off-line If off-line mount in read-only mode Use trusted tools Use MAC analysis Use File System Debugger UTSA IS 6353 Incident Response

Summary • Some say Unix Forensics are tougher than Windows Forensics • I agree the command usage is cryptic at time, but a tool is a tool • Tools presented are straight forward • Practice makes perfect UTSA IS 6353 Incident Response