Security event processing with SEC Simple Event Correlator

Security event processing with SEC (Simple Event Correlator) Risto Vaarandi Estonian Defence Forces

Outline • Event correlation – what it is and why do we need it? • Simple Event Correlator (SEC) – motivation, features and performance. • Sample event correlation scenarios. • SEC application experience and further reading. 2

Event correlation – what it is? • Commonly used formal definition (Jakobson and Weissman, 1995) – event correlation is an interpretation procedure where a new meaning is assigned to a set of events that occur within a predefined time interval. • A simple example – if a network link down event is not followed by network link up within 5 seconds, emit an output event network link broken; otherwise create an internal event network link short outage. If more than 10 network link short outage events have been observed within 3 hours, emit an output event network link quality degradation. • Please note that event correlation is a real-time (or nearreal-time) event processing technique! 3

Event correlation – why do we need it? • Event correlation produces events that are more meaningful to a human. • Event correlation can significantly reduce the number of events that are presented to the human. These considerations are important, because in today’s IT systems millions of events can occur in every hour – no hope for a human to follow the events and react to them in a timely manner! 4

Example - centralized event management infrastructure Central log server Notifications to the monitoring console event correlator Application event receiver Server events Router network Event logs Applications, servers, and network devices send their events to the central log server. Events are stored and correlated on the central log server, and output alerts from the event correlator are sent to the central monitoring console. 5

Simple Event Correlator (SEC) – motivation • Existing event correlation products are mostly heavyweight and expensive solutions: ü many products can only be deployed as a client-server system with a requirement for a dedicated high-end event correlation server, ü some products have been designed for specific tasks only (like network fault management), ü several products depend on other complex and expensive software packages (e. g. , Oracle), ü there are products that can’t be deployed without the assistance from vendor consultants! ü there exist only a few free and open-source event monitoring tools that have some support for event correlation. • SEC implements a completely different event correlation paradigm – lightweight and platform independent event correlation with an open source tool 6

SEC – design and features • Employs rule-based approach for event correlation – events are correlated according to rules written by a human analyst. • Lightweight design – the size of SEC is < 300 KB and < 9000 lines, runs as a single process. • Modest resource requirements – e. g. , with 10, 000+ event correlation operations and contexts active, < 20 MB memory is consumed. • Platform independence – SEC is written in Perl, thus runs on almost any UNIX flavor and Windows (with Cyg. Win Perl). • No dependencies on other software (Perl only). • Reads input events from one or more textual event logs (plugins can be written for binary formats). • Input events are matched with regular expressions, Perl subroutines, string patterns, and truth values. • Produces output by executing custom command lines (e. g. , sends an SNMP trap or an e-mail), writing to files and FIFOs, etc. • Licensing – GNU GPLv 2 (free and open-source software). 7

Snort IDS alert consolidation I # The following ruleset processes Snort IDS syslog messages: # Apr 4 10: 55 snorthost. mydomain [auth. alert] snort[18800]: # [1: 2528: 14] SMTP PCT Client_Hello overflow attempt [Classification: # Attempted Administrator Privilege Gain] [Priority: 1]: # {TCP} 192. 168. 5. 43: 28813 -> 192. 168. 250. 44: 25 # # If a Snort priority 1 alert has been seen for a certain source IP address, create # a context for this IP, in order to consolidate all events for the same source IP; # also, send a short note to the local admin that an attack has begun type=Single ptype=Reg. Exp pattern=snort[d+]: [[d: ]+]. *[Priority: 1]: S+ ([d. ]+): ? d* -> [d. ]+: ? d* context=!ATTACK_FROM_$1 continue=Take. Next desc=Priority 1 attack started from $1 action=create ATTACK_FROM_$1; pipe '%s' mail -s 'Snort: priority 1 attack from $1 (alert)' root 8

Snort IDS alert consolidation II # If a Snort priority 1 alert is observed for a certain source IP address, then: # 1) store it to the context for that IP (the context has been created by the previous rule) # 2) extend the context lifetime for 5 minutes. # # If the context lifetime ends (in other words, there have been no more events from # the given source IP), send all events from the context event store to the local admin # as a single mail. type=Single ptype=Reg. Exp pattern=snort[d+]: [[d: ]+]. *[Priority: 1]: S+ ([d. ]+): ? d* -> [d. ]+: ? d* context=ATTACK_FROM_$1 desc=Priority 1 incident from $1 action=add ATTACK_FROM_$1 $0; set ATTACK_FROM_$1 300 ( report ATTACK_FROM_$1 mail -s 'Snort: priority 1 attack from $1 (report)' root ) 9

User account probe detection I # Process sshd PAM authentication messages from Solaris system log, in order to detect repeated # probing of different user accounts: # Apr 3 14: 20: 19 myhost sshd[25888]: [ID 800047 auth. error] error: # PAM: Authentication failed for risto from myhost 2 # Apr 3 14: 20: 23 myhost sshd[25888]: [ID 800047 auth. info] Accepted # keyboard-interactive/pam for risto from 192. 168. 27. 69 port 9729 ssh 2 # # If a user fails to login with SSH and this is not followed by a successful login within 30 seconds, # generate an event LOGIN_FAILED_FOR_username type=Pair. With. Window ptype=Reg. Exp pattern=sshd[d+]: [ID d+ auth. error] error: PAM: Authentication failed for (S+) from S+ desc=PAM authentication failed for $1 action=event LOGIN_FAILED_FOR_$1 ptype 2=Reg. Exp pattern 2=sshd[d+]: [ID d+ auth. info] Accepted keyboard-interactive/pam for ($1) from S+ port d+ ssh 2 desc 2=PAM authentication successful for $1 action 2=logonly window=30 10

User account probe detection II # Count LOGIN_FAILED_FOR_username events from the previous rule – if 10 distinct usernames # have been seen within 10 minutes, alert the local admin; then switch to silent mode for 1 hour type=Single. With. Threshold ptype=Reg. Exp pattern=LOGIN_FAILED_FOR_(S+) context=!USER_$1_ALREADY_COUNTED && !COUNTING_OFF continue=Take. Next desc=Ten login failures for distinct users have been observed action=pipe '%s' mail -s 'PAM alert' root; create COUNTING_OFF 3600 window=600 thresh=10 # Set up a context for the username after it has been counted, in order to prevent repeated counting type=Single ptype=Reg. Exp pattern=LOGIN_FAILED_FOR_(S+) context=!USER_$1_ALREADY_COUNTED && !COUNTING_OFF desc=Set up the "count once" context for user $1 action=create USER_$1_ALREADY_COUNTED 600 11

User account probe detection III Feb 18 12: 10: 23 PAM: authentication failed for risto Feb 18 12: 10: 33 Accepted keyboard-interactive/pam for risto Pair. With. Window Feb 18 12: 15: 11 PAM: authentication failed for john Feb 18 12: 15: 56 ntpd: …. . doesn’t produce a match anymore at 12: 17: 19 Feb 18 12: 16: 49 PAM: authentication failed for john create: USER_john_ALREADY_COUNTED Single logonly: PAM authentication successful for risto (at 12: 10: 33) event: LOGIN_FAILED_FOR_john Single. With. Threshold 10 LOGIN_FAILED events for distinct users within 10 m pipe: send mail alert create: COUNTING_OFF (for getting 1 mail per 1 hour) neither USER_john_ALREADY_COUNTED nor COUNTING_OFF context exists at 12: 15: 41, so start the counting operation with the window of 12 10 minutes and set event counter to 1

SEC – performance • SEC’s event processing speed depends on a number of factors – the size and nature of the rule base, the nature of input data, and the underlying hardware. • Experiment 1: Linux workstation with a 3. 0 GHz P 4 CPU, one rule with a regular expression pattern, 1% of input lines were matching and written to a file by the rule – event processing speed was 17, 000+ events per second. • Experiment 2: Linux server with a 3. 2 GHz P 4 CPU, 100+ rules with complex regular expression patterns and nontrivial output actions (at least 2 -3 actions executed per second), banking card events with a complex structure as input – event processing speed was 3, 000 events per second. 13

SEC – experience • Has been developed by Risto Vaarandi for 8+ years. • Employed by many institutions – large North-American telecom companies, government and financial institutions, universities, medical and car industry, etc. • Has been used for security management, network management, system monitoring, etc. • Has been employed for correlating events from many sources – Snort IDS, HP Open. View, Cisco. Works, BMC Patrol, Nagios, syslogd/syslog-ng, various routers and firewalls, modems and other network equipment, etc. • Has been packaged for major Linux and BSD UNIX distributions – Red. Hat, Fedora, Debian, Ubuntu, Gentoo, Open. BSD, Free. BSD. • SEC mailing list has 300+ members. 14

SEC – references • SEC home page: http: //simple-evcorr. sourceforge. net. • John P. Rouillard, University of Massatchusetts at Boston, “Realtime Logfile Analysis Using the Simple Event Correlator (SEC)”, USENIX 2004 System Administration Conference (see also http: //www. cs. umb. edu/~rouilj/sec/). • Jeff Becklehimer and Cathy Willis, Cray Inc. and Josh Lothian, Don Maxwell, and David Vasil, Oak Ridge National Laboratory (ORNL), “Real Time Health Monitoring of the Cray XT Series Using the Simple Event Correlator (SEC)“, 2007 Cray User Group Conference (see also http: //www. nccs. gov/wpcontent/uploads/2007/08/becklehimer_paper 1. pdf). • James Turnbull, “Hardening Linux” (Chapter 5), Apress 2005 (see also http: //www. apress. com/book/view/1590594444). • Risto Vaarandi, “Simple Event Correlator for real-time security log monitoring”, Hakin 9 Magazine 1/2006 (6) (see also http: //en. hakin 9. org/attachments/pdf/hakin 9_05_2006_10_EN_str 2839. pdf). 15