Implementing Oblivious Hashing Using Overlapped Instruction Encodings Matthias

Implementing Oblivious Hashing Using Overlapped Instruction Encodings Matthias Jacob Nokia Research Mariusz H. Jakubowski Ramarathnam Venkatesan Microsoft Research ACM Multimedia and Security ‘ 07 Dallas, TX (USA) September 20 -21, 2007

Introduction • Field of work: Software protection – Obfuscation and tamper-resistance – Prevention (or delaying) of reverse engineering and hacking – Securing of content-rights systems (DRM) • Background: Two specific protection techniques – Oblivious hashing (OH): Computing hashes (“fingerprints”) of execution traces – Overlapped code: “Jumping into the middle of instructions” to obfuscate and protect against disassembly • Goals of our work: – Apply overlapped code towards obfuscation and tamperresistance via OH. – Study new techniques in terms of formal models, avoiding “ad hoc” approaches. ACM Multimedia and Security ’ 07 September 20 -21, 2007 2

Overview Oblivious hashing via overlapped code • Introduction • Background – Software protection – Oblivious hashing (OH) – Overlapped code • Code interleaving • Conclusion ACM Multimedia and Security ’ 07 September 20 -21, 2007 3

Software Protection • Obfuscation – Making programs “hard to understand” • Tamper-resistance – Making programs “hard to modify” • Obfuscation tamper-resistance • Tamper-resistance obfuscation? ACM Multimedia and Security ’ 07 September 20 -21, 2007 4

Formal Obfuscation • Impossible in general – Black-box model (Barak et al. ): Ø“Source code” doesn’t help adversary who can examine input-output behavior. – Worst-case programs and poly-time attackers • Possible in specific limited scenarios – Secret hiding by hashing (Lynn et al. ) – Point functions (Wee, Kalai et al. ) • Results difficult to use in practice. ACM Multimedia and Security ’ 07 September 20 -21, 2007 5

Tamper-Resistance • Many techniques used in practice – e. g. : – – Code-integrity checksums (e. g. , Atallah et al. ’s software guards) Anti-debugging and anti-disassembly methods Virtual machines and interpreters Polymorphic and metamorphic code • Never-ending battle on a very active field – Targets: DRM, CD/DVD protection, games, dongles, licensing, etc. – Defenses: Binary packers and “cryptors, ” special compilers, transformation tools, programming strategies, etc. • Current techniques tend to be “ad hoc: ” – No provable security – No analysis of time required to crack protected instances ACM Multimedia and Security ’ 07 September 20 -21, 2007 6

Tamper-Resistance Model Abstraction of software tamper-resistance (Dedić et al. , IH ’ 07) • Program: A graph G • Execution: A “random” walk on G • Integrity checks: – Probabilistic monitoring of a set of G’s nodes – Detection of failures that lead to delayed responses • Security analysis: “Graph game” on G between attacker and defender • OH and overlapped code in context of model: – Provide a source of integrity checks. – Help enforce “local indistinguishability” and other engineering assumptions about implementation. ACM Multimedia and Security ’ 07 September 20 -21, 2007 7

Oblivious Hashing • Computation of hashes over program traces – Initialize hash values at specific points. – Update hashes upon assignments and branches. INITIALIZE_HASH(hash 1); int x = 123; UPDATE_HASH(hash 1, x); int x = 123; if (Get. User. Input() > 10) { x = x + 1; } else { printf("Hellon"); } Original code Hash transform if (Get. User. Input() > { UPDATE_HASH(hash 1, x = x + 1; UPDATE_HASH(hash 1, } else { UPDATE_HASH(hash 1, printf("Hellon"); } 10) BRANCH_ID_1); x); BRANCH_ID_2); VERIFY_HASH(hash 1); Hashed code ACM Multimedia and Security ’ 07 September 20 -21, 2007 8

Overlapped Code • Code sharing among different paths – Semantic: Sharing of code blocks among execution paths. – Physical: Sharing of code bytes among machine or byte -code instructions. • Purposes – Anti-disassembly and anti-decompilation – Obfuscation – Tamper-resistance from code sharing and explicit OH ACM Multimedia and Security ’ 07 September 20 -21, 2007 9

Semantic Overlap increase_win() { increase_ctr(&win); return win; } Code section is shared along different paths: return win; ACM Multimedia and Security ’ 07 increase_loss() { increase_ctr(&loss) return loss; } increase_ctr(*ctr) { (*ctr)++; } Automated via code outlining return loss; September 20 -21, 2007 10

Physical Overlap Execution and disassembly depend on entry point into code. Sample x 86 code: B 8 04 05 2 D 05 90 Offset 0: B 8 04 05 2 D 05 90 Offset 3: mov eax, 2 D 0504 B 8 sub eax, 90 Offset 1: B 8 04 05 2 D 05 90 add eax, 90052 D Offset 4: mov eax, 52 D 0504 nop Offset 2: 04 05 2 D 05 90 sub eax, 9005 Offset 5: add al, 5 sub eax, 9005 05 90 sub eax, 90 Note: Disassembly tends to resynchronize naturally – but we can prevent this. ACM Multimedia and Security ’ 07 September 20 -21, 2007 11

Disassembly Synchronization • Often observed in practice, but previously not explained mathematically. • Limits effectiveness of code overlapping for security. • Requires explicit anti-synchronization measures to enforce protection. • Rigorous explanation: Kruskal count Example of corruption and synchronization: Corrupted byte Synchronization point ACM Multimedia and Security ’ 07 00411410 00411411 00411413 00411419 0041141 A 0041141 B 0041141 C 55 8 B EC 81 EC C 0 00 00 00 53 56 57 8 D BD 40 FF FF FF 00411410 00411411 00411413 00411415 00411418 0041141 B 0041141 C 55 8 B 12 C 0 00 57 8 D push mov sub push lea push EC mov EC adc 00 00 rol 53 56 add push BD 40 FF FF FF lea ebp, esp, 0 C 0 h ebx esi edi, [ebp-0 C 0 h] ebp, esp ch, ah byte ptr [eax], 0 byte ptr [ebx+56 h], dl edi, [ebp-0 C 0 h] September 20 -21, 2007 12

Disassembly Synchronization • Disassembly: A “leapfrog” process over code bytes – Each byte address contains an instruction of a definite length. – After disassembling an instruction, a disassembler skips to the next instruction. • Example: Sequence of instruction lengths at consecutive offsets: 34626345335427314 Synchronization point 34626345335427314 Sequence of instruction lengths Disassembly at offset: Kruskal count: Such disassembly synchronizes in about B 2/16 steps, where B = average # of bytes per instruction. ACM Multimedia and Security ’ 07 0 1 2 3 4 3 2 3 4 6 6 3 3 4 4 5 September 20 -21, 2007 14 14 14 13

Disassembly Synchronization Model of the disassembly process • Let Instruction. Length(address) = length of instruction found at address. • Starting at “slightly different” addresses x and y, a disassembler iterates: x x + Instruction. Length(x) y y + Instruction. Length(y) (“leapfrog x”) (“leapfrog y”) • Our goal: Compute N = approximate number of steps before any intermediate x is equal to any intermediate y. • Treat all possible values of x-y as states of a Markov chain. • N is the coupling time of this Markov chain. • Kruskal count: N is about B 2/16, where B is the average instruction length. ACM Multimedia and Security ’ 07 September 20 -21, 2007 14

Code Interleaving • A method to overlap arbitrary code blocks – Explicitly prevents disassembly resynchronization – Adds tamper-resistance • Hash of instruction bytes only (like traditional code checksums) • Hash of instruction bytes and program state (like oblivious hashing) • Basic algorithm – Code interspersing: Create a block of interleaved instructions from two code blocks. – Code merging: Inject hashing instructions overlapped with existing instructions. ACM Multimedia and Security ’ 07 September 20 -21, 2007 15

Code Interleaving: Basic Idea SEQ 1: INST_1 INST_2 SEQ 2: INST_A INST_B Two input code blocks ACM Multimedia and Security ’ 07 September 20 -21, 2007 16

Code Interleaving: Basic Idea SEQ 1: INST_1 INST_2 SEQ 2: INST_A INST_B Two input code blocks l SEQ 1: INST_1 JMP L 2 SEQ 2: INST_A JMP LB L 2: INST_2 JMP L 3 LB: INST_B L 3: After code interspersing Code interspersing: Interleave instructions, injecting jumps as needed to maintain control flow. ACM Multimedia and Security ’ 07 September 20 -21, 2007 17

Code Interleaving: Basic Idea SEQ 1: INST_1 INST_2 SEQ 2: INST_A INST_B SEQ 1: INST_1 JMP L 2 SEQ 2: INST_A JMP LB L 2: INST_2 JMP L 3 LB: INST_B L 3: SEQ 1: INST_1 HASH_1 INST_2 HASH_2 Disassembly at SEQ 1 SEQ 2: INST_A HASH_A INST_B Disassembly at SEQ 2 Two input code blocks After code interspersing After code merging l l l Code interspersing: Interleave instructions, injecting jumps as needed to maintain control flow. Code merging: Replace jumps with hash instructions, maintaining control flow. o E. g. : JMP L 2; INST_A; JMP_LB transforms into HASH_1 contains INST_A and part of HASH_A. Suitable hash instructions must be found (and fit together like puzzle pieces). o Various possibilities identified on x 86. o Can also design custom byte-codes to maximize utility of overlapping. ACM Multimedia and Security ’ 07 September 20 -21, 2007 18

Code Interleaving: Example SEQ 1: C 1 E 0 02 I 11: 40 C 3 shl eax, 2 inc eax ret SEQ 2: 48 I 21: C 1 E 8 03 C 3 dec eax shr eax, 3 ret C 1 E 0 02 EB 03 shl eax, 2 jmp I 11 SEQ 2: 48 EB 04 I 11: 90 40 EB 03 I 21: C 1 E 8 03 O: 90 C 3 Two input code blocks (x 86) dec jmp nop inc jmp shr nop ret eax I 21 eax O eax, 3 After code interspersing SEQ 1: SEQ 2: C 1 E 0 02 81 F 1 48 81 E 9 90 I 11: 40 81 C 1 E 8 03 90 O: C 3 shl xor inc add ret eax, 2 ecx, 90 E 98148 eax ecx, 9003 E 8 C 1 48 81 E 9 90 40 81 C 1 I 21: C 1 E 8 03 O: 90 C 3 Disassembly at SEQ 1 dec eax sub ecx, C 1814090 shr eax, 3 nop ret Disassembly at SEQ 2 After code merging (OH instructions in red) ACM Multimedia and Security ’ 07 September 20 -21, 2007 19

Code Interleaving • Observations – Tamper-resistance comes from two main sources: • Implicit: Shared instruction bytes • Explicit: OH instructions – Disassembly synchronization is explicitly prevented. – Method enables code-byte hashes even on architectures that do not allow explicit access to code bytes. • Extensions – Iteration to build up complexity • Enhances security at little or no implementation cost. • Complex (emergent) code patterns and behaviors can arise. – Implementation over custom byte codes designed to maximize utility of overlapping (unlike x 86) ACM Multimedia and Security ’ 07 September 20 -21, 2007 20

Experimental Results Performance impact on Spec. INT benchmarks: 0 = no overlapping, 1 = full overlapping • • • Tool implementation using Vulcan (binary-rewriting framework) Reasonable impact on performance, depending on desired security level Remaining work on analyzing security in practice ACM Multimedia and Security ’ 07 September 20 -21, 2007 21

Conclusion • Contributions – Investigation of overlapped code for software protection • • Study of disassembly synchronization and other roadblocks Design of code interleaving and outlining to address limitations Integrity checking via oblivious hashing Placement in context of security models, not ad hoc methods – Tool implementations to verify practical effectiveness • Code interleaving and outlining for x 86 binaries • Iteration framework to enhance security • Future work – Security analysis in theory and practice – Other overlapped-code methods – Porting to custom byte-codes ACM Multimedia and Security ’ 07 September 20 -21, 2007 22