NETWORK TRAFFIC ANALYSIS OPTIMIZATION FOR SIGNATUREBASED INTRUSION DETECTION

NETWORK TRAFFIC ANALYSIS OPTIMIZATION FOR SIGNATURE-BASED INTRUSION DETECTION SYSTEMS Dmitry S. Kazachkin student, Computational systems lab at CMC MSU Dennis Y. Gamayunov scientific advisor, Ph. D, Computational systems lab at CMC MSU

INTRODUCTION High-performance traffic analysis at gigabit channels Optimization of a given signature set matching Packet – the unit of analysis Operations number minimization for a single packet

SIGNATURE FORMAL DEFINITION (1) Packet Header space – pair {H, P} (H) – {H 1, H 2, . . . , Hn}, Hi belongs to finite Payload State (P) – text line of unrestricted length vector (C) – {С 1, С 2, . . . , Сk}, Ci belongs to finite space

SIGNATURE FORMAL DEFINITION (2) Header condition – logical predicate; Hi – arguments Payload condition – set of functions: Cond. Pi(P, C) — logical predicate; payload and state vector are arguments Effectsi = {Effecti, j(P, C)} — represents side effect on variable Сi, performed on Cond. Pi(P, C) success Reaction – element of event classes finite set

SIGNATURE FORMAL DEFINITION (3) Signature – triplet <SH, SP, R>, where SH – set of header conditions SP – ordered set of payload conditions R – reaction. RES(<SH, SP, R>, H, P, C): {R}, if all the header conditions return ‘true’ and all the payload conditions successfully evaluate consecutive. Æ, else

SNORT SIGNATURES STRUCTURE Action – action performed on rule activation (usually, ‘alert’) Header – context-independent conditions on packet header Options – context-dependent conditions on packet payload Info – rule info and message, generated at rule activation

SNORT SIGNATURES FORMALIZATION (1) Header H 1 — protocol type (TCP / UDP / ICMP / other IP) H 2 and H 3 — IP and port of packet source H 4 — packet direction (from server/to server) H 5 and H 6 — IP and port of packet destination Payload – packet payload byte string State vector – C = {C 1}, C 1 – position of text marker

SNORT SIGNATURES FORMALIZATION (2) Cond. Hk(H) – logical expression, operating {Hi} only <Cond. Pk(P, C), Effectk(P, C)> – pair of functions representing pattern matching performed after marker Reaction – finite set of identifiers, which represent action and info sections

R-LANGUAGE FORMAL SIDE Automaton - (K, PS, T, PT, s 0, I, g q) S – states set; PS – state predicates set; T – transitions set; PT – transition conditions set; s 0 – initial state; I – automaton exemplars set; g – global environment; q – global timer queue automaton exemplar – series of slices NK x L x Q x E NK – current state; L – local environment; Q – local timer queue; E – local events queue

R-LANGUAGE DECLARATIVE SIDE Scenario Events subscription Variables declaration – scenario fields Functions declaration – scenario methods States declaration One is marked as initial Transitions declaration Transition type – consuming/nonconsuming/unwinding Condition – logical function Body – code block

SIGNATURE CONVERSION TO R-LANG (1) S. T. Eckmann, "Translating Snort rules to STATL scenarios“ Typified events on every packet Src. IP, Src. Port, Direction, Dst. IP, Dst. Port Payload Scenario field variable Current Header conditions – logical expressions on {Hi} Payload conditions – boolean function calls, with side-effect on Current Reactions – alert sending code blocks

SIGNATURE CONVERSION TO R-LANG (2) scenario sc(<event corresponding to packet type> ev) { <Secondary variables definition> initial state st 0; consuming transition st 0 ->st 0 event <event corresponding to packet type> ( <Header condition 1> && … && <Header condition N> ){ <Restoring state vector> if(<Payload condition 1>) … if(<Payload condition M>) <Reaction>; } };

HEADER-BASED OPTIMIZATION (1) Conditions alternative — a set of pairs <SPi, Ri>, where SPi – ordered set of payload conditions Ri — reaction Alternative-containing signature – a pair <SH, SA>, where SH — set header conditions SA — condition alternative. Set of signatures with the same header are converted to alternativecontaining signature this way: Alternative-containing signature evaluation result is defined as:

HEADER-BASED OPTIMIZATION (2) R-lang transition body of the alternative-containing signature model: <Restoring state vector> if(<Series of payload conditions 1>) <Reaction 1>; <Restoring state vector> if(<Series of payload conditions 2>) <Reaction 2>; … <Restoring state vector> if(<Series of payload conditions N>) <Reaction N>; We get the same RES set with less operation by grouping signatures with the same header conditions set. 519 different header condition sets for Snort base of 6372 signatures grants a good performance speed-up

HEADER-BASED OPTIMIZATION (3) Tree-style order provides even more economy. consuming transition st 0 ->st 0 event <event corresponding to packet type>(true){ if(<Header condition 1>) { if(<Header condition 2>) <predicate alternative for header granted by 1, 2> if(<Header condition 3>) if(<Header condition 4>) <predicate alternative for header granted by 1, 3, 4> } } Snort analysis engine use that header optimization only, that does not allow achieving further speed-up on a fixed signature set.

PREDICATE TREE Fact: Condition chains contained in conditions alternative can have the same beginnings. Predicate tree ST – tree: the edges are marked with a payload condition the nodes are marked with a reaction set, possibly empty the leafs are marked with non-empty reaction sets

PREDICATE TREE EVALUATION Predicate tree evaluation recursive traversal of tree from root node conditions control sub-trees evaluation reaction performing if current node is marked context restoration after sub-tree evaluation RES(ST, P, C) – {all the reaction achieved} Statement: Let predicate tree ST is converted from condition alternative SA. Then RES(ST, P, C)=RES(SA, P, C).

ADDITIONAL OPTIMIZATIONS Synonymic bunch – set of leafs: the same {R} the same edges parent PCRE union Aho-Corasick algorithm

STATIC RESULT ANALYSIS Tree profit = edges number in the tree - total number of payload conditions Relational tree profit = tree profit / total number of payload conditions For 2 signature groups (Samba): ~2000 signatures tree profit > 11000 relational tree profit > 80% Overall relational tree profit ~62%. 40 pairs of identical signatures

DYNAMIC RESULT ANALYSIS N seconds of flood with legal packets, a bad packet after them Seconds of alert delay Before Optimization After optimization Test 1, 5 seconds 123 10 Test 1, 20 seconds 391 24 Test 2, 5 seconds 103 37 Test 2, 20 seconds 203 133 Test 1 – legal traffic is ignored by most rules FTP “HELP” command Test 2 – legal traffic is checked by lots of rules specially constructed HTTP request

CONCLUSION Proposed methods provides significant speed-up of packet analysis at experimental IDS. Though speed-up is significant, it is still not enough. There is a need for other optimizations, e. g. parallel signature matching.

REFERENCES 1. Snort IDS, http: //www. snort. org 2. S. T. Eckmann, "Translating Snort rules to STATL scenarios" presented at the 4 th International Symposium on Recent Advances in Intrusion Detection (RAID 2001), Davis, CA, October 2001, LNCS 2212, pp. 69 -84. 3. M. Norton, "Optimizing Pattern Matching for Intrusion Detection, " white paper, Sourcefire Inc. , 2004 [Online] Avaliable: http: // docs. idsresearch. org/Optimizing. Pattern. Matching/For. IDS. pdf. 4. D. U. Gamayunov, “Network objects behavior analysis for detecting computer attacks” Ph. D thesis, Faculty of Computational Math and Cybernetics, Moscow State University, Moscow, 2007. 5. S. T. Eckmann, G. Vigna, and R. A. Kemmerer. “STATL: An Attack Language for State-based Intrusion Detection” Dept. of Computer Science, University of California, Santa Barbara, 2000. 6. G. Vigna, R. Kemmerer, "Net. STAT: A Network-based Intrusion Detection Approach. " Proceedings of the 14 th Annual Computer Security Application Conference, Scottsdale, Arizona, December 1998. 7. M. Roesch. "Writing Snort Rules: How To write Snort rules and keep your sanity" [Online] Avaliable: http: //www. snort. org.

Thank you for attention