Software Reverse Engineering SRE SRE Introduction 1 SRE

Software Reverse Engineering (SRE) SRE Introduction 1

SRE q Software Reverse Engineering q Can be used for good. . . q …or not-so-good o Also known as Reverse Code Engineering (RCE) o Or simply “reversing” o Understand malware o Understand legacy code (design recovery) o Remove usage restrictions from software o Find and exploit flaws in software o Cheat at games, etc. SRE Introduction 2

SRE q For now, we assume… o Reverse engineer is an attacker o Attacker only has exe (no source code) q Attacker might want to o Understand the software o Modify the software q SRE usually focused on Windows o So here we’ll focus on Windows SRE Introduction 3

SRE Tools q Disassembler q Debugger q Hex Editor q Regmon, Filemon, VMware, etc. o Converts exe to assembly as best it can o Cannot always disassemble correctly o In general, it is not possible to re-assemble disassembly into working exe o Must step thru code to completely understand it o Labor intensive lack of automated tools o To patch (make permanent changes to) exe file SRE Introduction 4

SRE Tools q IDA Pro is the top-rated disassembler q Soft. ICE is “alpha and omega” of debuggers q Olly. Dbg is a high-quality shareware debugger q Hex editor to view/modify bits of exe q Regmon, Filemon freeware o Cost is a few hundred dollars o Converts binary to assembly (as best it can) o Cost is in the $1000’s o “Kernel mode” debugger o Can debug anything, even the OS o Includes a good disassembler o Ultra. Edit is good freeware o HIEW useful for patching exe SRE Introduction 5

Why is a Debugger Needed? q Disassembler gives static results o Good overview of program logic o User must “mentally execute” program o Difficult to jump to specific place in the code q Debugger is dynamic o Can set break points o Can treat complex code as “black box” o Not all code disassembles correctly q Disassembler and debugger both required for any serious SRE task SRE Introduction 6

SRE Necessary Skills Working knowledge of target assembly code q Experience with the tools q o IDA Pro sophisticated and complex o Soft. ICE large two-volume users manual o Olly. Dbg best choice for this class Knowledge of Windows Portable Executable (PE) file format q Boundless patience and optimism q SRE is a tedious, labor-intensive process! q SRE Introduction 7

SRE Example We consider a very simple example q This example only requires disassembler (IDA Pro) and hex editor q o Trudy disassembles to understand code o Trudy also wants to patch the code q For most real-world code, also need a debugger (Olly. Dbg) SRE Introduction 8

SRE Example Program requires serial number q But Trudy doesn’t know the serial number! q q Can Trudy get the serial number from exe? SRE Introduction 9

SRE Example q IDA Pro disassembly q Looks like serial number is S 123 N 456 SRE Introduction 10

SRE Example q Try the serial number S 123 N 456 q It works! q Can Trudy do better? SRE Introduction 11

SRE Example q Again, q And IDA Pro disassembly hex view… SRE Introduction 12

SRE Example q test eax, eax does AND of eax with itself o Flag bit set to 0 only if eax is 0 o If test yields 0, then jz is true Trudy wants jz to always be true! q Can Trudy patch exe so jz always holds? q SRE Introduction 13

SRE Example q Can Trudy patch exe so that jz always true? xor Assembly test eax, eax xor eax, eax SRE Introduction jz always true!!! Hex 85 C 0 … 33 C 0 … 14

SRE Example q Edit serial. exe with hex editor serial. exe serial. Patch. exe q Save as serial. Patch. exe SRE Introduction 15

SRE Example q Any “serial number” now works! q Very convenient for Trudy! SRE Introduction 16

SRE Example q Back to IDA Pro disassembly… serial. exe serial. Patch. exe SRE Introduction 17

SRE Attack Mitigation Impossible to prevent SRE on open system q But can make such attacks more difficult q Anti-disassembly techniques q o To confuse static view of code q Anti-debugging techniques q Tamper-resistance q Code obfuscation o To confuse dynamic view of code o Code checks itself to detect tampering o Make code more difficult to understand SRE Introduction 18

Anti-disassembly q Anti-disassembly methods include o o q Encrypted or packed object code False disassembly Self-modifying code Many other techniques Encryption prevents disassembly o But need decrypted code to decrypt the code! o Same problem as with polymorphic viruses SRE Introduction 19

Anti-disassembly Example q Suppose inst 1 jmp q What actual code instructions are junk inst 3 inst 4 … the disassembler sees inst 1 inst 2 inst 3 inst 4 inst 5 inst 6 … q This is example of “false disassembly” q Persistent attacker will figure it out! SRE Introduction 20

Anti-debugging q Monitor for o Use of debug registers o Inserted breakpoints q Debugger might not handle threads well o Interacting threads may confuse debugger Many other debugger-unfriendly tricks q Undetectable debugger possible in principle q o Hardware-based debugging (Hard. ICE) is possible SRE Introduction 21

Anti-debugger Example inst 1 inst 2 inst 3 inst 4 inst 5 inst 6 q … Suppose when program gets inst 1, it prefetches inst 2, inst 3 and inst 4 o This is done to increase efficiency Suppose when debugger executes inst 1, it does not pre-fetch instructions q Can we use this difference to confuse the debugger? q SRE Introduction 22

Anti-debugger Example junk 4 inst 5 inst 6 inst 1 inst 2 inst 3 inst … Suppose inst 1 overwrites inst 4 in memory Then program (without debugger) will be OK since it fetched inst 4 at same time as inst 1 q Debugger will be confused when it reaches junk where inst 4 is supposed to be q Problem for program if this segment of code executed more than once! q Also, code is very platform-dependent q Again, persistent attacker can figure this out! q q SRE Introduction 23

Tamper-resistance Goal is to make patching more difficult q Code can hash parts of itself q If tampering occurs, hash check fails q Research has shown can get good coverage of code with small performance penalty q But don’t want all checks to look similar q o Or else easier for attacker to remove checks q This approach sometimes called “guards” SRE Introduction 24

Code Obfuscation Goal is to make code hard to understand q Opposite of good software engineering! q o For example, spaghetti code q Much research into more robust obfuscation o Example: opaque predicate int x, y : if((x y) > (x x 2 x y+y y)){…} o The if() conditional is always false q Attacker wastes time analyzing dead code SRE Introduction 25

Code Obfuscation Code obfuscation sometimes promoted as a powerful security technique q Diffie and Hellman’s original conjectures for public key crypto based on code obfuscation! q Recently it has been shown that obfuscation probably cannot provide “strong” security q o On the (im)possibility of obfuscating programs o But some question these result (Thomborson) q Obfuscation might still have practical uses! o Even if it can never be as strong as crypto SRE Introduction 26

Authentication Example Software used to determine authentication q Ultimately, authentication is 1 -bit decision q o Regardless of method used (pwd, biometric, …) Somewhere in authentication software, a single bit determines success/failure q If attacker can find this bit, he can force authentication to always succeed q Obfuscation makes it more difficult for attacker to find this all-important bit q SRE Introduction 27

Obfuscation forces attacker to analyze larger amounts of code q Method could be combined with q o Anti-disassembly techniques o Anti-debugging techniques o Code tamper-checking All of these increase work (and pain) for attacker q But a persistent attacker can ultimately win q SRE Introduction 28