Polymorphism and IDS Black Hat Briefings Las Vegas

Polymorphism and IDS Black Hat Briefings Las Vegas 2001 Chad R. Skipper Sr. Software Engineer Symantec Corp. 1

whoami Chad R. Skipper • Air Force - systems counter intelligence, OSI investigations, information warfare, and exploit intelligence • Trident Data Systems – Network/Sys/Security Administrator • L-3 Network Security/Symantec – Sr. Software Engineer • Signature Development • IDS Evasion Techniques cskipper@symantec. com 2

Overview • Evolution of malicious polymorphic code • Paradigm shift • Polymorphic coding • ADMmutate by K 2 • http: //www. ktwo. ca/ • TCPDumps • IDS Response 3

Polymorphism What is polymorphism • The ability to appear in many forms • Continuous change (unique coding) • Independent of encryption • Morphs regexp’s within attacks • Can exist on multiple platforms 4

Evolution of Polymorphism Simple Viruses • Replicates itself and is the easiest to detect • Virus always makes an exact replica of itself • Detection: Scan for a sequence of bytes found in the virus 5

Evolution of Polymorphism Encrypted Viruses • Response to detection was encrypting viruses • Hide the fixed bytes by encrypting the virus 6

Evolution of Polymorphism Encrypted Viruses • Consists of a virus decryption routine and an encrypted virus body • Uses encryption keys, but decryption remained constant, thus detection was a sequence of bytes of the decryption routine 7

Evolution of Polymorphism Encrypted Viruses • • 8 Executes decryption routine Gains control of the system Decrypts and gives control to virus Infection occurs Copies itself Encrypts itself Attaches itself to a new program

Evolution of Polymorphism Polymorphic Virus • Response to detection was polymorphism • Contains the encrypted body and decryption routine • Adds a mutation engine that generates randomized decryption routines with each use • Mutation engine and virus body are both encrypted • Result is the virus body encryption and decryption routines vary from infection to infection • NO FIXED SIGNATURE 9

Evolution of Polymorphism Polymorphic Virus • • 10 Decrypts virus and mutation engine Transfers control to the virus Copies itself and the mutation engine Invokes the mutation engine Randomly generates decryption routine Virus is now unique from the prior virus Attaches to a new program

Evolution of Polymorphism Problems with Polymorphic Virus Detection • Dark Avenger and Mt. E • Produces random programs • Billions-upon-billions of variations Polymorphic Virus Detection • One-by-one, line-by-line (Don’t think so) Generic Decryption • Slow Heuristic-Based Generic Decryption • Heuristic guesses • False Negatives 11

Evolution of Polymorphism Polymorphic Virus Detection Solutions • • • 12 Does not rely on heuristic guesses Relies on rules or profiles specific to each virus Rules out possibilities first Runs file in virtual machine (VM) Looks for triggers

Evolution of Polymorphism Polymorphic Virus Detection Solutions • • Load file into self-contained VM Is this file. exe, . com, . sys…? If. exe then A, B, C, D, and E are virus behaviors Suspect files • • • A, B, C A, B, D D, B, E • Observes A, then “D, B, E” are out • Observes B, then remaining are still in • Observes D, then “A, B, C” are out and “A, B, D” are in 13

The Paradigm Shift Concepts used from Polymorphic Viruses • • • Mutation engine Polymorphic algorithm Morphing of the payload to include • • 14 Shell code NOP’s Encoder/Decoder Non-Operational Padding

The Paradigm Shift The intent of Polymorphic Attacks • To evade signature analysis of IDS • Signature analysis looks at • • • 15 Shell code NOP’s Specific offsets within a payload ASCII Headers

Encoding Process Shell code • Morphed prior to launch with each subsequent morphing unique • ROT, MOV • XOR (exclusive-or) Randomly generated value • 0 xor 0 = 0 • 0 xor 1 = 1 • 1 xor 0 = 1 • 1 xor 1 = 0 • If the first or the second operand, but not both, is one, the result is one; otherwise the result is zero. 16

Encoding Process Shell code • Randomly generated xor value of 0 x 23 • DNS – Snort alert UDP $EXTERNAL any -> $INTERNAL 53 (msg: "IDS 489/named-exploit-tsig-lsd"; content: "|3 F 909090 EB 3 B 31 DB 5 F 83 EF 7 C 8 D 7710 897704 8 D 4 F 20|"; classtype: system-attempt; reference: arachnids, 489; ) 17

Encoding Process Shell code • Shell code of: 0 x 3 F 909090 EB 3 B 31 DB 5 F 83 EF 7 C 8 D 7710 897704 8 D 4 F 20 • XOR with the value of 0 x 23 • We get: 0 x 1 C B 3 B 3 B 3 C 818 12 F 8 7 C A 0 CC 5 F AE 5433 AA 5427 AE 6 C 03 • This can give us over 64, 000 permutations for 1 byte • BTW, the computational overhead for this for NIDS may/will be substantial. 18

Encoding Process NOP’s • No operation assembly processor instruction • So, we substitute known NOP’s with other characters that do not affect the outcome of the code 19

Encoding Process NOP’s • Platform specific NOP’s • • • 20 AIX – 0 x 4 ffffb 82 Digital – 0 x 47 ff 041 f HP – 0 x 0 b 390280 Intel – 0 x 90 SGI – 0 x 240 f 1234 SPARC – 0 x 13 c 01 ca 6; 0 xa 61 cc 013, 0 x 801 c 4011

Encoding Process NOP’s • Substitutional NOP’s per K 2 • Intel • 0 x 49 • 0 x 4 b • 0 x 45 • SPARC • • • 21 0 xa 21 c 8012 0 xb 606401 a 0 xa 026 e 042

Encoding Process Encoder/Decoder • My first thought was that we can detect the Encoder/Decoder • “It would not be cool if the IDS vendor could simply detect our decoder. ” - K 2 • FAT CHANCE… This would be too easy • Techniques used are multiple code paths, nonoperational padding, and randomly generated instructions • Decoder processes the data after the overflow 22

Attacks 23

TCPDumps (Normal) 4500 04 e 8 0 a 0 a 2 a 05 8018 7 d 78 008 c 3 e 60 9090 9090 (Cut) 9090 ffc 3 5 e 31 0 c 31 c 050 50 b 0 17 e 8 0853 8 d 1 e c 40 c e 8 bb ffff 7 c 6 b 0408 24 be 81 0933 b 342 9090 4000 0019 0000 9090 4006 70 e 1 0101 9090 0 f 4 c 3 dc 3 080 a 9090 0 a 0 a ad 03 0400 9090 2 a 2 a 63 b 0 22 e 1 9090 9090 c 089 b 08 d d 2 ff 895 e ffff 7 c 6 b 9090 46 b 4 e 8 df ffff 0853 ff 2 f 0408 eb 48 8846 ffff 83 c 4 b 03 b 6269 7 c 6 b 9 aff b 988 ff 83 0431 e 8 bb 6 e 2 f 0408 ffff 4607 c 404 c 050 ffff 7368 7 c 6 b ff 07 8946 31 c 0 8 d 5 e ff 83 ffff 0408

TCPDumps (Polymorphed) 4500 04 e 8 0 a 0 a 2 a 05 8018 7 d 78 008 c 5 fe 5 4 d 49 414 b 4827 494 a 414 d 4 c 4 c 4244 414 b 4 c 4 d 454 d (CUT) 36 aa 763 c 9383 e 886 c 046 85 c 0 c 087 e 8 c 5 55 b 5 6207 a 647 fc 66 fac 6 2 bde a 225 b 128 6 e 7 a d 48 d 5589 c 3 c 9 25 be 81 0933 b 342 4949 4845 434 c 4 c 27 4540 f 54 d 4000 0019 0000 4 b 49 4 bf 5 4 b 4 d 494 c 4940 404 d 4006 70 e 1 0101 494 d 484 d 4 af 9 4 a 49 f 54 c 4 d 27 0 f 4 c 3 dc 3 080 a 4 df 5 4 b 4 d f 54 a 4140 4945 f 94 b 0 a 0 a ad 03 0400 40 f 9 4549 4 c 4 d 414 d 40 f 5 4 d 4 d 2 a 2 a 63 b 0 4475 4040 4449 274 c 48 f 5 4 b 42 5 b 31 9640 46 e 2 ffff 6 aff 1 afb 7889 2328 5589 d 61 d c 9 b 0 968 c e 685 ff 7 e 7 a 82 d 4 e 9 c 3 c 9 3465 c 319 0408 df 6 a c 08 c c 085 413 e 2230 5589 29 b 2 1 a 4 d c 81 f 816 b 1866 e 083 c 0 eb a 6 c 9 85 be c 3 b 5 3807 d 48 d 5219 0408 5993 c 601 0 bb 0 5589 ec 71 6 e 72 6 a 26 5589 d 91 e 816 b 3106 f 533 346 b c 331 b 570 0 df 6 b 168 c 3 b 5 c 3 c 9 0408

TCPDumps (Polymorphed) 26 4500 04 e 8 0 a 0 a 2 a 05 8018 7 d 78 008 c e 181 40 f 9 414 b 404 b 484 b f 941 4449 f 545 4 b 40 f 5 f 5 2742 454 d 42 f 9 (CUT) 5896 83 c 0 4 a 40 6 a 18 aa 8 c c 083 eb 06 e 8 c 9 fe 95 e 1 a 9 0 d 67 7 fc 8 51 e 6 a 870 0905 3286 c 55 a 5723 fea 9 4067 be 81 0933 b 342 454 b 444 b 4 af 9 4327 4027 f 54 b f 548 4000 0019 0000 4449 4 c 44 4 b 4 a 4 d 44 2745 4 c 41 4 d 45 4006 70 e 1 0101 444 a 4845 f 94 d 48 f 5 41 f 5 4 b 4 c 0 f 4 c 3 dc 3 080 a 4040 4 d 40 404 a 45 f 9 f 549 4927 f 545 0 a 0 a ad 03 0400 4342 4944 2740 4149 f 544 444 b 4442 2 a 2 a 63 b 0 c 60 f 4 af 9 f 948 f 94 b 4341 4 d 4 a 4941 424 d 4 a 68 5889 c 601 ffff c 1 df b 1 db d 3 a 9 8808 fea 9 7 d 3 d 9801 c 193 9640 ffd 9 f 92 c 5747 4067 b 7 cb 40 b 7 0408 56 bf 3106 96 c 1 ea 1 e 8910 fea 9 8292 b 16 d 633 f cb 6 b 5 b 31 9346 c 0 ed 2567 0610 401 b bba 9 5723 d 1 b 7 0408 c 091 f 946 e 2 e 9 fea 9 4751 c 552 c 106 fea 9 723 e cb 6 b c 1 e 8 8 cc 0 409 f 36 de 8 e 58 32 c 6 401 b 4067 0408

Network Intrusion Response Protocol Analysis • Application Layer 27

Network Intrusion Response Protocol Analysis • What protocol is it? • IP, IPX… • • 28 If IP then is it TCP, UPD, ICMP… If TCP then is it HTTP, DNS, FTP… If HTTP then apply HTTP signatures Determine if alert is needed

Network Intrusion Response Protocol Analysis • Break the payload down into manageable parts • Look for expected results • Anything out of that range – alert Abnormal HTTP Normal HTTP 29

Network Intrusion Response Protocol Analysis • Can detect polymorphic attacks • Proactive • Better performance • Harder to evade • May be possible to create polymorphic code that looks like normal traffic on some services 30

Network Intrusion Response Pattern Matching • Searches for set patterns within packets, such as shell-code, NOP’s, and ASCII • Pattern matching is defeated by polymorphic attacks 31

Network Intrusion Response Snort Example – Pattern Matching DNS - Snort alert UDP $EXTERNAL any -> $INTERNAL 53 (msg: "IDS 489/named-exploit-tsig-lsd"; content: "|3 F 909090 EB 3 B 31 DB 5 F 83 EF 7 C 8 D 7710 897704 8 D 4 F 20|"; classtype: system-attempt; reference: arachnids, 489; ) TFN - Snort alert ICMP any -> any (msg: "IDS 425/ddos-tfn 2 kicmp_possible_communication"; itype: 0; icmp_id: 0; content: "AAAAA"; classtype: system-success; reference: arachnids, 425; ) 32

Network Intrusion Response Snort Example – Pattern Matching X 86 NOP’s - Snort alert UDP $EXTERNAL any -> $INTERNAL any (msg: "IDS 362/shellcode-x 86 -nops-udp"; content: "|90 90 90 90 90 90 90|"; classtype: systemattempt; reference: arachnids, 362; ) 33

Network Intrusion Response Binary Signatures • Detecting binary strings within protocols such as SMTP • Attacks against text-only services could check for characters outside the standard text range • FTP • Could pick up polymorphic attacks 34

Network Intrusion Response Packet Size • Detecting unusual large amounts of data streams • POP 3, RPC, HTTP, FTP • Can pick up polymorphic attacks 35

Network Intrusion Response Connection Time • Abnormal connection time rates such as lengthy DNS collaboration • DNS, HTTP, RPC, etc… • Time based • Expensive • Could detect polymorphic attacks by timing the session between hosts 36

Network Intrusion Response Outcome Detection – Success/Failure • Able to detect response to attacks • Able to detect “/bin/sh” leaving on port 53 • Could detect polymorphic attacks • Another evasion technique is the response from the victim being hashed/encrypted/scrambled 37

Network Intrusion Response Outcome Detection – Success/Failure • Solaris snmp. Xdmid - LAST STAGE OF DELIRIUM • NOP’s to server • 00 1 C 00 00 00 40 00 00 00 11 FF FF FF 80 00 00 00 1 C 00 • /bin/ksh to server • 00 08 00 00 00 2 F 00 00 00 62 00 00 00 69 00 00. . . /. . . b. . . i. . • 00 6 E 00 00 00 2 F 00 00 00 6 B 00 00 00 73 00 00. n. . . /. . . k. . . s. . • 00 68 00 00 04 00 00 00. h. . . • uname –a to server • 82 8 C 2 F 62 69 6 E 2 F 75 6 E 61 6 D 65 20 2 D 61 0 A. . /bin/uname -a. 38

Network Intrusion Response Outcome Detection – Success/Failure • Solaris snmp. Xdmid - LAST STAGE OF DELIRIUM • Response to uname –a • 2 E E 1 53 75 6 E 4 F 53 20 73 61 2 D 73 6 F 6 C 61 72. . Sun. OS sa-solar 69 73 2 D 30 32 20 35 2 E 38 20 47 65 6 E 65 72 69 is 02 5. 8 Generi 63 20 73 75 6 E 34 75 20 73 70 61 72 63 20 53 55 c sun 4 u sparc 4 E 57 2 C 55 6 C 74 72 61 2 D 35 5 F 31 30 0 A NW, Ultra-5_10. • Response to /etc/passwd • 35 1 C 72 6 F 6 F 74 3 A 78 3 A 30 3 A 31 3 A 53 75 70 5. root: x: 0: 1: Sup 65 72 2 D 55 73 65 72 3 A 2 F 73 62 69 6 E 2 F er-User: /: /sbin/ 73 68 0 A 64 61 65 6 D 6 F 6 E 3 A 78 3 A 31 3 A sh. daemon: x: 1: 1: 0070: 3 A 2 F 3 A 0 A 62 69 6 E 3 A 78 3 A 32 3 A 3 A 2 F : /: . bin: x: 2: 2: : / 39

Network Intrusion Response Log Analysis • Event Viewer, /var/adm/messages/, /var/log/syslog, etc. • Able to detect abnormal occurrences within the host • Can detect polymorphic attacks • # more /var/adm/messages • May 25 11: 55: 09 sa-solaris-02 dmispd: [ID 922709 daemon. error] One instance of this daemon is already running on this machine 40

Host Intrusion Response Access/Change Analysis • Changes to any audited file • Spawning of child processes • Removal of any audited file • Replacement of any audited file • Can detect polymorphic attacks 41

Host Intrusion Response Port Activity • Unusual port activity • RPC – ttdb – active session to outside host • Could detect polymorphic attacks as they occur 42

Defeating Polymorphic Attacks 43

Assessment and Intrusion Detection (IDS) Host-Based Inspect system configuration files, password files for weak passwords, and other system objects for policy violations Network-Based Reenact common intrusion or attack scenarios ID and report network vulnerabilities and suggest corrective actions “Proactive” (scheduled) Assessment Monitor audit and log data Active “sensors” on servers and workstations monitor user actions and protect resources, applications, and data 44 “Reactive” Collect information from the network for real-time monitoring (24 x 7) IDS

Future trends from the past State of NIDS detection is where Anti-Virus was in mid 90’s IDS Evasion is now just getting started Polymorphic Virus Stats (SARC www. sarc. com) • 1988 - The first virus with variable key encryption (between infections) • 1990 - Polymorphic viruses found in the United States including V 2 Px, Virus-90 and Virus-101 viruses • 1992 – First polymorphic engine that could be plugged into a virus as an add-on • Today - ~2, 000 – 5, 000 polymorphic viruses today (Not all in the wild) 45