Intrusion Detection Somesh Jha University of Wisconsin J

Intrusion Detection Somesh Jha University of Wisconsin J. Giffin and S. Jha

Intrusion Detection Goal: Discover attempts to maliciously gain access to a system 2/11/2022 J. Giffin and S. Jha 2

Network Intrusion Detection Systems (NIDS) • Inspects packets at certain vantage points – For example, behind the routers • Look for malicious or anomalous behavior • Much more fine-grained than firewalls – Example: drop a packet whose payload “matches” a certain string 2/11/2022 J. Giffin and S. Jha 3

Classification of NIDS • Signature-based – Also called misuse detection – Establish a database of malicious patterns – If a sequence of packets “matches” one of the patterns, raise an alarm – Positives • Good attack libraries • Easy to understand the results – Negatives • Unable to detect new attacks or variants of old attacks • Example – Snort, Bro, NFR, … 2/11/2022 J. Giffin and S. Jha 4

Classification of NIDS • Anomaly-based – Establish a statistical profile of normal traffic – If monitored traffic deviates “sufficiently” from the established profile, raise an alarm • Positives – Can detect new attacks • Negatives – High false alarm rate – Intruder can go under the “radar” • Examples – Mostly research systems 2/11/2022 J. Giffin and S. Jha 5

Classification of NIDS • Stateless – Need to keep no state – Example: raise an alarm if you see a packet that contains the pattern “mellissa” • Positives – Very fast • Negatives – For some attacks need to keep state 2/11/2022 J. Giffin and S. Jha 6

Classification of NIDS • Stateful – Keeps state – Sometime need to do reassembly • Reassemble packets that belong to the same connection, e. g. , packets that belong to the same ssh session • Quite hard! (out-of-order delivery) – Positives • Can detect more attacks – Negatives • Requires too much memory 2/11/2022 J. Giffin and S. Jha 7

Snort Filtered packet stream 2/11/2022 J. Giffin and S. Jha 8

libpcap • Takes the “raw” packet stream • Parses the packets and presents them as a – Filtered packet stream • Library for packet capture • Website for more details – http: //www-nrg. ee. lbl. gov/. 2/11/2022 J. Giffin and S. Jha 9

Malicious Pattern Example alert tcp any -> 10. 1. 1. 0/24 80 (content: “/cgi-bin/phf”; msg: “PHF probe!”; ) action pass log alert 2/11/2022 protocol source address source port J. Giffin and S. Jha destination address destination port 10

Malicious Patterns Example • content: “/cgi-bin/phf” – Matches any packet whose payload contains the string “/cgi-bin/phf” – Look at http: //www. cert. org/advisories/CA 1996 -06. html • msg: “PHF probe!” – Generate this message if a match happens 2/11/2022 J. Giffin and S. Jha 11

More Examples alert tcp any -> 10. 1. 1. 0/24 6000: 6010 (msg: “X traffic”; ) alert tcp !10. 1. 1. 0/24 any -> 10. 1. 1. 0/24 6000: 6010 (msg: “X traffic”; ) 2/11/2022 J. Giffin and S. Jha 12

How to generate new patterns? • Buffer overrun found in Internet Message Access Protocol (IMAP) – http: //www. cert. org/advisories/CA-199709. html • Run exploit in a test network and record all traffic • Examine the content of the attack packet 2/11/2022 J. Giffin and S. Jha 13

Notional "IMAP buffer overflow" packet 052499 -22: 27: 58. 403313 192. 168. 1. 4: 1034 -> 192. 168. 1. 3: 143 TCP TTL: 64 TOS: 0 x 0 DF ***PA* Seq: 0 x 5295 B 44 E Ack: 0 x 1 B 4 F 8970 Win: 0 x 7 D 78 90 90 90 90 EB 3 B. . . . ; 5 E 89 76 08 31 ED 31 C 9 31 C 0 88 6 E 07 89 6 E 0 C ^. v. 1. 1. 1. . n. B 0 0 B 89 F 3 8 D 6 E 08 89 E 9 8 D 6 E 0 C 89 EA CD 80. . . n. . 31 DB 89 D 8 40 CD 80 90 90 90 1. . . @. . . 90 90 90 E 8 C 0 FF FF FF. . . . 2 F 62 69 6 E 2 F 73 68 90 90 90 /bin/sh. . 2/11/2022 J. Giffin and S. Jha 14

Alert rule for the new buffer overflow alert tcp any -> 192. 168. 1. 0/24 143 (content: "|E 8 C 0 FFFF FF|/bin/sh"; msg: "New IMAP Buffer Overflow detected!"; ) Can mix hex formatted bytecode and text 2/11/2022 J. Giffin and S. Jha 15

Advantages of Snort • Lightweight – Small footprint – Focused monitoring: highly tuned Snort for the SMTP server • Malicious patterns easy to develop – Large user community – Consider the IRDP denial-of-service attack – Rule for this attack available on the same day the attack was announced 2/11/2022 J. Giffin and S. Jha 16

Disadvantages • Does not perform stream reassembly – Attackers can use that to “fool” Snort – Break one attack packet into a stream • Pattern matching is expensive – Matching patterns in payloads is expensive (avoid it!) • Rule development methodology is adhoc 2/11/2022 J. Giffin and S. Jha 17

Host-based ID • Monitor interaction between a specific program and OS • Raise an alarm if suspicious “system calls” are observed • Unlike NIDS monitoring happens at the end hosts • Need to model – Unusual behavior – Normal behavior 2/11/2022 J. Giffin and S. Jha 18

Goal: Discover attempts to maliciously gain access to a system Misuse Detection Specification-Based Monitoring Anomaly Detection • Specify patterns of attack or misuse • Specify constraints upon • Learn typical behavior of application program behavior • Ensure misuse patterns do not arise at runtime • Variations indicate • Ensure execution does not violate specification potential intrusions • Snort • Our work; Ko, et. al. • IDES • Rigid: cannot adapt to novel attacks • Specifications can be cumbersome to create • High false alarm rate 2/11/2022 J. Giffin and S. Jha 19

Specification-Based Monitoring • Two components: – Specification: Indicates constraints upon program behavior – Enforcement: How the specification is verified at runtime or from audit data 2/11/2022 J. Giffin and S. Jha 20

Analyst or Administrator Training Sets Static Source Code Analysis Static Binary Code Analysis Specification Enforcement Execution Matches Model of Application Execution Obeys Static Ruleset 2/11/2022 J. Giffin and S. Jha 21

Representative Work by Ko, et al. • Specification: Programmers or administrators specify correct program behavior PROGRAM fingerd read(X) : - worldreadable(X); bind(79); write(“/etc/log”); exec(“/usr/ucb/finger”); END • Enforcement: At runtime, only allow actions that match the specified policy 2/11/2022 J. Giffin and S. Jha 22

Analyst or Administrator Training Sets Static Source Code Analysis Static Binary Code Analysis Specification Enforcement Execution Matches Model of Application Execution Obeys Static Ruleset 2/11/2022 J. Giffin and S. Jha 23

Representative Work by Forrest, et al • Specification: Learn correct program behavior with training – Record sequences of system calls • Enforcement: Only accept behaviors similar to learned patterns • Example system: STIDE 2/11/2022 J. Giffin and S. Jha 24

Training • Repeatedly run the program, varying the input • For some n, record all sequences of n system calls observed – n depends upon the program • End result: database of n-tuples of system calls 2/11/2022 J. Giffin and S. Jha 25

cat (print file contents) geteuid, getegid, getgid, fstat, open, fstat, lseek, mmap, read, memcntl, write, lseek, munmap, lseek, close, exit 2/11/2022 geteuid, getuid, getegid, getgid, fstat, open / lseek open, fstat lseek, mmap / munmap / close mmap, read, memcntl, write, lseek munmap, lseek close, close / exit J. Giffin and S. Jha 26

Enforcement • Monitor system calls generated by application • Ensure that the last n calls match a sequence in the database • Option: Allow slight deviation from database – Training set may have been incomplete 2/11/2022 J. Giffin and S. Jha 27

cat (print file contents) geteuid, getegid, getgid, fstat, open, fstat, lseek, mmap, read, memcntl, write, lseek, munmap, lseek, close, exit Accepts incorrect system call sequences geteuid, getegid, getgid, fstat, lseek, close, exit 2/11/2022 geteuid, getuid, getegid, getgid, fstat, open / lseek open, fstat lseek, mmap / munmap / close mmap, read, memcntl, write, lseek munmap, lseek close, close / exit J. Giffin and S. Jha 28

Drawbacks • Accepts incorrect call sequences – Due to window-based approach with ambiguity – Opportunity for attack sequence to go undetected • Only learn behaviors exercised in training set – Not all execution paths followed • Users must construct valid training sets • Users must determine window size 2/11/2022 J. Giffin and S. Jha 29

Drawbacks • Specification may overfit the data • If training on real data, training set may contain exploits – Learn exploit pattern as normal 2/11/2022 J. Giffin and S. Jha 30

Analyst or Administrator Training Sets Static Source Code Analysis Static Binary Code Analysis Specification Enforcement Execution Matches Model of Application Execution Obeys Static Ruleset 2/11/2022 J. Giffin and S. Jha 31

Our Approach function: save %sp, 0 x 96, %sp cmp %i 0, 0 bge L 1 mov 15, %o 1 call read mov 0, %o 0 call line nop b L 2 nop L 1: call read mov %i 0, %o 0 call close mov %i 0, %o 0 L 2: ret restore 2/11/2022 Specification: Static analysis of binary code • Specifications are automatically generated • Not reliant upon programmers to produce accurate specifications • Analyzes all execution paths • Source code may be unavailable J. Giffin and S. Jha 32

Our Approach read Enforcement: Operate an automaton modeling correct system call sequences • Dynamic ruleset close 2/11/2022 line • More expressive than static ruleset of Ko, et. al. J. Giffin and S. Jha 33

Non-Deterministic Finite Automaton (NFA) • Structure read close 2/11/2022 read line – States – Labeled edges between states • Edge labels are input symbols – call names • Path to any accepting state defines valid sequence of calls J. Giffin and S. Jha 34

Our Approach read Enforcement: Operate an automaton modeling correct system call sequences • Dynamic ruleset close 2/11/2022 line • More expressive than static ruleset of Ko, et. al. J. Giffin and S. Jha 35

The Binary View (SPARC) function: save %sp, 0 x 96, %sp cmp %i 0, 0 bge L 1 mov 15, %o 1 call read mov 0, %o 0 call line nop b L 2 nop L 1: call read mov %i 0, %o 0 call close mov %i 0, %o 0 L 2: ret restore 2/11/2022 function (int a) { if (a < 0) { read(0, 15); line(); } else { read(a, 15); close(a); } } J. Giffin and S. Jha 36

Control Flow Graph Generation CFG ENTRY function: save %sp, 0 x 96, %sp cmp %i 0, 0 bge L 1 mov 15, %o 1 call read mov 0, %o 0 call line nop b L 2 nop L 1: call read mov %i 0, %o 0 call close mov %i 0, %o 0 L 2: ret restore 2/11/2022 bge call read call close call line ret CFG EXIT J. Giffin and S. Jha 37

Control Flow Graph Translation CFG ENTRY bge read close read line call read call close call line ret CFG EXIT 2/11/2022 J. Giffin and S. Jha 38

Control Flow Graph Translation CFG ENTRY bge read close read line call read call close call line ret CFG EXIT 2/11/2022 J. Giffin and S. Jha 39

A Interprocedural Model Generation read close line 2/11/2022 J. Giffin and S. Jha 40

A Interprocedural Model Generation read close line 2/11/2022 line write J. Giffin and S. Jha 41

A Interprocedural Model Generation read close line 2/11/2022 line write J. Giffin and S. Jha B line close 42

A read close 2/11/2022 Interprocedural Model Generation read line write J. Giffin and S. Jha B line close 43

A read close 2/11/2022 Interprocedural Model Generation read B line write J. Giffin and S. Jha close 44

Possible Paths A read close 2/11/2022 read B line write J. Giffin and S. Jha close 45

Possible Paths A read close 2/11/2022 read B line write J. Giffin and S. Jha close 46

Impossible Paths A read close 2/11/2022 read B line write J. Giffin and S. Jha close 47

Impossible Paths A read close 2/11/2022 read B line write J. Giffin and S. Jha close 48

A read Adding Context Sensitivity read line B Y X write close Y close X 2/11/2022 J. Giffin and S. Jha 49

PDA State Explosion • ε-edge identifiers maintained on a stack – Stack may grow to be unbounded X • Solution: – Bound the maximum size of the runtime stack – A regular language overapproximation of the context-free language of the PDA 2/11/2022 J. Giffin and S. Jha 50

Data Flow Analysis function: save %sp, 0 x 96, %sp cmp %i 0, 0 bge L 1 mov 15, %o 1 call read mov 0, %o 0 call line nop b L 2 nop L 1: call read mov %i 0, %o 0 call close mov %i 0, %o 0 L 2: ret restore 2/11/2022 Argument recovery • Statically known arguments constrain remote calls • Reduces opportunity given to attackers J. Giffin and S. Jha 51

Call Site Renaming function: save %sp, 0 x 96, %sp cmp $i 0, 0 bge L 1 mov 15, %o 1 call read mov 0, %o 0 call line nop b L 2 nop L 1: call read mov %i 0, %o 0 call close mov %i 0, %o 0 L 2: ret restore 2/11/2022 • Give each monitored call site a unique name • Associates arguments with call sites • Obfuscation • Reduces nondeterminism J. Giffin and S. Jha 52

Call Site Renaming function: save %sp, 0 x 96, %sp cmp $i 0, 0 bge L 1 mov 15, %o 1 call _638 mov 0, %o 0 call line nop b L 2 nop L 1: call read mov %i 0, %o 0 call close mov %i 0, %o 0 L 2: ret restore 2/11/2022 • Give each monitored call site a unique name • Associates arguments with call sites • Obfuscation • Reduces nondeterminism J. Giffin and S. Jha 53

Call Site Renaming function: save %sp, 0 x 96, %sp cmp $i 0, 0 bge L 1 mov 15, %o 1 call _638 mov 0, %o 0 call line nop b L 2 nop L 1: call _83 mov %i 0, %o 0 call close mov %i 0, %o 0 L 2: ret restore 2/11/2022 • Give each monitored call site a unique name • Associates arguments with call sites • Obfuscation • Reduces nondeterminism J. Giffin and S. Jha 54

Call Site Renaming function: save %sp, 0 x 96, %sp cmp $i 0, 0 bge L 1 mov 15, %o 1 call _638 mov 0, %o 0 call line nop b L 2 nop L 1: call _83 mov %i 0, %o 0 call _1920 mov %i 0, %o 0 L 2: ret restore 2/11/2022 • Give each monitored call site a unique name • Associates arguments with call sites • Obfuscation • Reduces nondeterminism J. Giffin and S. Jha 55

Call Site Renaming read close line 2/11/2022 • Give each monitored call site a unique name • Associates arguments with call sites • Obfuscation • Reduces nondeterminism J. Giffin and S. Jha 56

Call Site Renaming _638 _83 _1920 line 2/11/2022 • Give each monitored call site a unique name • Associates arguments with call sites • Obfuscation • Reduces nondeterminism J. Giffin and S. Jha 57

Technical Challenges • Integrating other specification sources • Optimal null call insertion • C++ vtable analysis 2/11/2022 J. Giffin and S. Jha 58