Introduction to Reverse Engineering Inbar Raz Malware Research

Introduction to Reverse Engineering Inbar Raz Malware Research Lab Manager Information Security – Theory vs. Reality Tel Aviv University, 368 -4474 -01, Winter 2012 -2013 Lecture 7 © 2012 Check Point Software Technologies Ltd.

What is Reverse Engineering? Reverse engineering is the process of discovering the technological principles of a device, object, or system through analysis of its structure, function, and operation. aka: Reversing, RE, SRE © 2012 Check Point Software Technologies Ltd. 2 2

Why do it? Discover Trade Secrets Find Vulnerabilities Academic Research (Yeah, right…) Circumvent [Copy] Protection Patch Binary and Alter Behavior Pure Curiosity © 2012 Check Point Software Technologies Ltd. Analyse Protocols 3 3

Sounds awesome, right? © 2012 Check Point Software Technologies Ltd. 4 4

So where’s the catch? § Low-level is, well, low level… 00401000 push ebp 00401001 mov ebp, esp 00401003 push ecx 00401004 push ecx 00401005 and dword ptr [ebp-4], 0 00401009 push esi 0040100 A mov esi, [ebp+8] for (Serial = 0, i = 0; i < strlen(User. Name); 0040100 D i++) push { edi Cur. Char = (int) User. Name[i]; 0040100 E push esi Serial += Cur. Char; 0040100 F call ds: [00402008 h] Serial = (((Serial << 1) && 0 x. FFFFFFFE) ((Serialedi, >> 31) 00401015|| mov eax && 1)); Serial = (((Serial * Cur. Char) + Cur. Char) ^ Cur. Char); 00401017 xor edx, edx } 00401019 test edi, edi User. Serial = ~((User. Serial ^ 0 x 1337 C 0 DE) 0 x. BADC 0 DE 5); 0040101 B jle 00401047 h 0040101 D movsx ecx, byte ptr [edx+esi] 00401021 add [ebp-4], ecx 00401024 mov [ebp-8], ecx 00401027 rol dword ptr [ebp-4], 1 0040102 A mov eax, ecx 0040102 C imul eax, [ebp-4] 00401030 mov [ebp-4], eax 00401033 mov eax, [ebp-8] 00401036 add [ebp-4], eax 00401039 xor [ebp-4], ecx 0040103 C inc edx 0040103 D cmp edx, edi 0040103 F jl 0040101 Dh © 2012 Check Point Software Technologies Ltd. 5 5

So where’s the catch? § Low-level is, well, low level… § Needle in a haystack – Average opcode size: 3 bytes – Average executable size: 500 KB (on Win. XP) – There are executables, libraries, drivers…. © 2012 Check Point Software Technologies Ltd. 6 6

So where’s the catch? § Low-level is, well, low level… § Needle in a haystack § Sometimes, the code resists § Sometimes, the code fights back – Detect reversing tools – Detect VMs and emulators © 2012 Check Point Software Technologies Ltd. 8 8

A Battle of Wits § Author writes code § Reverser reverses it § Author

So what do you need in order to be a good reverser? © 2012

We’ll come back to this… © 2012 Check Point Software Technologies Ltd. 11 11

Tools of the Trade § Disassembler (Static code analysis) § Debugger (Dynamic code analysis)

Disassemblers © 2012 Check Point Software Technologies Ltd. 13 13

The old world: Sourcer © 2012 Check Point Software Technologies Ltd. 14 14

The old world: Sourcer © 2012 Check Point Software Technologies Ltd. 15 15

Old ages: Sourcer © 2012 Check Point Software Technologies Ltd. 16 16

Old ages: Sourcer © 2012 Check Point Software Technologies Ltd. 17 17

Welcome to Windows: W 32 DASM © 2012 Check Point Software Technologies Ltd. 18

The Holy Grail: IDA-Pro § Started as an Interactive Dis-Assembler, enabling user interaction with the disassembler’s decisions. § Slowly evolved into an automatic RE tool: – Built-in full-control script language – Library recognition (including user-generated) – Function prototype information – Display – Propagate throughout the code – Support for plug-ins – Support for Python scripting – Multi-architecture, cross-platform support – Full incorporation with built-in and external debuggers © 2012 Check Point Software Technologies Ltd. 19 19

Debuggers בדיבאג זין – בדיזיין באג © 2012 Check Point Software Technologies Ltd.

First, there was DEBUG… © 2012 Check Point Software Technologies Ltd. 21 21

GUI and much more: Turbo Debugger © 2012 Check Point Software Technologies Ltd. 22

GUI and much more: Turbo Debugger © 2012 Check Point Software Technologies Ltd. 23

GUI and much more: Turbo Debugger © 2012 Check Point Software Technologies Ltd. 24

Next major step: Soft-ICE © 2012 Check Point Software Technologies Ltd. 25 25

And finally: Olly. Dbg © 2012 Check Point Software Technologies Ltd. 26 26

Other Tools © 2012 Check Point Software Technologies Ltd. 27 27

Hex-Editor © 2012 Check Point Software Technologies Ltd. 28 28

PE Analyzer © 2012 Check Point Software Technologies Ltd. 29 29

Resource Editor © 2012 Check Point Software Technologies Ltd. 30 30

Methodology © 2012 Check Point Software Technologies Ltd. 31 31

How we do it? § Finding the interesting part – System calls (User mode and Kernel mode API) – Strings and constants – Dynamic loading of libraries – Provocation – Zoom-in and Zoom-out § Better and quicker (and how not to get stuck) – Iterative passes over the code – Leave no stone unturned – Macros/Scripts/Plugins – Bin. Diffing (manual and with tools) © 2012 Check Point Software Technologies Ltd. 32 32

Let’s play with them tools… © 2012 Check Point Software Technologies Ltd. 33 33

60 seconds on x 86 registers § General purpose registers: 32 bit/16 bit/8 bit § Index registers: 32 bit/16 bit § Segment registers: 16 bit § Flags: 32 bit/16 bit © 2012 Check Point Software Technologies Ltd. 34 34

Exercise 1: Static Reversing © 2012 Check Point Software Technologies Ltd. 35 35

Exercise 1: Static Reversing § Target: a 2004 “Crack-Me” § Tools: IDA-Pro © 2012

Exercise 2: Dynamic Reversing © 2012 Check Point Software Technologies Ltd. 37 37

Exercise 2: Dynamic Reversing § Target: a 2004 “Crack-Me” § Tools: Olly. Dbg, IDA-Pro

Exercise 3: Simple Anti-Debugging © 2012 Check Point Software Technologies Ltd. 39 39

Exercise 3: Simple Anti Debugging § Target: a 2006 “Crack-Me” § Tools: Olly. Dbg

Reversing Malware § Malware is comprised of the following building blocks: – Infection Vector – Concealment – Operation – Communications § Check Point’s Anti-Malware Software Blade sits at the gateway § Therefore, communications interest us the most © 2012 Check Point Software Technologies Ltd. 41 41

Introducing: Spy Eye § A Crime. Ware Tool. Kit, originating in Russia. § Used mostly for stealing financial information, but will settle for any other identity information and key logging… § Like any serious trojan, Spy Eye compresses its traffic and encrypts it – Compression is performed using a public library (LZO) – Encryption algorithm is proprietary © 2012 Check Point Software Technologies Ltd. 42 42

Act 1: Encryption © 2012 Check Point Software Technologies Ltd. 43 43

Act 2: Configuration Download © 2012 Check Point Software Technologies Ltd. 44 44

Act 3: Another Encryption © 2012 Check Point Software Technologies Ltd. 45 45

What makes a good reverser? Qualities Knowledge • Patient • Assembly Language • Curious • Some High-Level programming • Best: origin of binary • Persistent • Outside-the-Box Thinking • Operating System Internals • API • Data Structures • File Structures • Good scripting skills • Anti-Debugging Tricks • Optional: Good lookin’ © 2012 Check Point Software Technologies Ltd. 47 47

Outside-the-Box Thinking © 2012 Check Point Software Technologies Ltd. 48 48

And remember, kids: Binary Reverse Engineer + © 2012 Check Point Software Technologies Ltd.

Which means… A F L I © 2012 Check Point Software Technologies Ltd. 50

Questions? © 2012 Check Point Software Technologies Ltd. 51 51

Check your Opportunities We are looking for: Excellent Computer Science/Software Engineering students for a variety of developing positions § Talented software developers § Security & Malware researchers § To learn more about our openings please check our website: www. checkpoint. com © 2012 Check Point Software Technologies Ltd. 52 52