Arc Sight Flex Connectors Introduction to the Arc

Arc. Sight: Flex. Connectors Introduction to the Arc. Sight Flex. Connector Framework Till Jäger, Solution Architect CISSP © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

What this is (and what it is not) This is not • A SQL Training • A Regex Training • A Flex. Connector Training So what is it then? • Broad overview of functionality • Guide how to approach your problem • Help to get started 2 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Agenda • • • 3 Connector basics Flexconnectors basics Tools Regex parsers WUC parsers DB Connectors Syslog basics Connector by example: Syslog Categorization Tips & tricks / Best practice Parser overriding © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Smart. Connector Basics © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

What is a Smart. Connector? A Smart. Connector is software that collects events from end-point devices, normalizes the events, send them to an Arc. Sight destination. 5 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Smart. Connector Event Flow 6 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Smart. Connector Types • • • 7 File Connectors Database Connectors Scanner Connectors API Connectors SNMP Connectors Microsoft Windows Event Log Connectors Syslog Connectors Flex. Connectors Model Import Connectors Net. Flow Connector © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Smart. Connector Directory Structure /current/bin – executables and scripts Arcsight. bat / arcsight. sh /current/config/agent – default/base configurations Agent. defaults. properties Agent. wrapper. conf. base /current/logs – Smart. Connector generated logs Agent. log. n Agent. out. wrapper. n /current/user /agent 8 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Smart. Connector Directory Structure /current/user/agent – Connector properties and destination-specific configurations, preserve state files Agent. properties Hosts. txt Persisted. properties <agentid>. xml config files Syslog. properties /current/user/agentdata – queue, cache, persistence files. prstdout. n files cache. dflt. n files Syslogd. n files /current/user/agent/aup – directory for content aup, additional data mappings, zones Destination specific directories Additional data mapping files – ngadatamapping. properties /current/user/agent/acp – directory for categorization files Categorization files – content aup 9 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Smart. Connector Directory Structure /current/user/agent/fcp – directory for parser overrides Parser overrides /current/user/agent/flexagent – directory for custom parsers, and regex tester Custom parsers /current/user/agent/map – directory for map. n. properties files Map. n. properties 10 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Smart. Connector Configuration Files Location of the Configuration files: /current/user/agent/ agent. properties • contains the global configuration Agent<id>. xml • contains the destination specific configuration 11 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Smart. Connector Configuration Files agent. wrapper. conf Service configuration file that includes the wrapper configuration • E. g. increasing the heap size, extending ping timeouts, . . agent. defaults. properties Located in /current/config/agent Contains the default framework parameters Contains syntax for enabling debugging and increasing agent log file size, and agent log count 12 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Additional Data Mappings / map. x. properties Additional Data Mappings • • • Often not all fields are explicitly mapped to schema fields Mapped to additional data Can be mapped on-demand from Console • Right-click on the Smart. Connector, and send the command “get additional data names” map. x. properties • • • useful for adding additional details to events Uses “getters” and “setters” For example: When source user name is John, set Device Custom String 1 to John’s phone 408. 555. 1234. event. source. User. Name, set. event. device. Custom. String 1 John, 408. 555. 1234 • 13 /current/user/agent/map © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Categorization • Arc. Sight Content • Provides additional meaning to the event • Requires 3 fields to be populated to work: Device Event Class ID: Device Product: Device Vendor • Special kind of map file in user/agent/acp/categorizer/current • Required for Foundation Content / System Content to work properly 14 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Log Analysis If anything goes wrong: /logs/agent. log Search for ERROR For log trend analysis use Log. Fu • Log. Fu is an Arc. Sight log Analysis Utility • Part of every Smart. Connector install • Open a CMD Prompt CD to /current/logs logs>. . binarcsight agent logfu -a 15 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Log Analysis: Log. Fu! 16 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Flex. Connectors Choose The Right Type © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Before You Get Started • • 18 Read the Flexconnector Guide Familiarize with the Event Schema Download latest Connector build Get to know the basics of Regex © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Flex. Connector Types • • 19 File Reader Multi Folder Reader (Batch and/or Realtime) Syslog Subagent SNMP Database Vulnerability Scanner Model Import WUC (Windows Unified Connector) parser © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Flex. Connector Parser Types Parser File name for each parser type has a unique extension • Delimited Log Parser (sdkfilereader) • Regex Parser (sdkrfilereader) • Key Value parser (sdkkeyvaluefilereader) • Database parser (sdktbdatabase, sdkibdatabase) • SNMP parser (sdksnmp. X. snmptrap) • XML parser (xqueryparser) • A combination of those (more later) 20 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Log File Flex Connectors Log File Characteristics Number of files • Single file or multiple files? • Do we know the exact file names? • If file names change, do they have an expressible pattern? Access to the files • Do we have access to the files locally or remotely? • If not they may have to be copied periodically Static or Dynamic Data • Is data in the file growing? • Are the files static and generated by the device or copied from the device periodically? 21 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Log File Flex Connectors Log File Characteristics What kind of data is in the file? Delimited Data Free Form Data Key Value Pairs XML 22 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Choose The Right Type 23 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Did You Know that… • You can re-use a connector install by deleting the agent. properties and run „. /arcsight agentsetup“ again • You can copy an installed connector to a new location and re-use it • A Connector usually doesn‘t need Administrative/root access • You don‘t need a running ESM/logger to test your Connector 24 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Flex. Connectors Structure of a properties file © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Configuration File comments. start. with=# delimiter=, contains. empty. tokens=true token. count=9 event. name=Action event. message=__concatenate(Action, ”; ”, Message) event. device. Receipt. Time=Event_Time event. attacker. Address=Source. IP token[0]. name=Firewall. IP event. attacker. Port=Source. Port token[0]. type=IPAddress event. target. Address=Destination. IP token[1]. name=Event_Time event. target. Port=Destination. Port event. transport. Protocol=Protocol token[1]. type=Time. Stamp token[1]. format=yyyy/MM/dd HH: mm: ssevent. device. Address=Firewall. IP event. device. Severity=Action token[2]. name=Source. IP event. device. Custom. Number 1=__regex. Token(Message, [^. ]. ? token[2]. type=IPAddress Rule (\d+)) token[3]. name=Source. Port event. device. Custom. Number 1 Label=__string. Constant(“Rule token[3]. type=Integer Number”) token[4]. name=Destination. IP token[4]. type=IPAddress token[5]. name=Destination. Port severity. map. high. if. device. Severity=drop token[5]. type=Integer severity. map. low. if. device. Severity=accept token[6]. name=Protocol token[6]. type=String token[7]. name=Action 26 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Double Underscore Operators More listed in the flex developers guide Operator Meaning Example __string. Constant() Specifies a constant string for event mapping __string. Constant(“This is the Event Name”) __concatenate() Concatenates two or more fields __concatenate(Token 1, Token 2) __regex. Token() Used to parse a field one time __regex. Token($1, User ([^)]))(. *)) __to. Upper. Case() Converts a string to all upper case __to. Upper. Case(Token. Name) __create. Time. Stamp Creates a timestamp from a date field and a time field __create. Time. Stamp(Date, Time) 27 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Flex. Connectors Tools © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Tools • • • 29 Flex. Connector wizard for delimited logs Arc. Sight Regex Tool Flexcon wizard in Conapp Notepad++ We!Analyze Netsend Tail for Windows Regex. Buddy SQuirre. L SQL Client © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Tools: Regex Buddy + Most comprehensive tool + Many Regex dialects, great library + Easy to performance optimize expression - commercial 30 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Tools For JDBC Connectors: SQuirre. L • If SQuirre. L works, your connector will work! • Test Connectivity, Credentials, Drivers, Connection String, etc! 31 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Flex. Connectors Regex Parsers © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Regex Configuration File Parser Configuration Common regular expression to match all entries in the log file 2. 2. 2. 1: FW 2002/10/01 18: 12 3. 3: 24356 2. 2: 80 (tcp) action=drop Message: Rule 25 2. 2. 2. 1: VPN 2002/10/01 18: 12 4. 4: 36542 2. 2: 80 (tcp) action=drop Message: Encryption failed, username jsmith Rule 5 regex=(d+. d+): (S+) (d+/d+ d+: d+) (d+. d+): (d+) ((S+)) action=(S+) Message: s(. *)? Rules+(d+) Token Declaration Each value in parentheses will be tokenized: token. count=11 token[5]. name=Destination. IP token[0]. name=Firewall_IP token[5]. type=IPAddress token[0]. type=IPAddress token[6]. name=Destination. Port token[1]. name=Process. Name token[6]. type=Integer token[1]. type=String token[7]. name=Protocol token[2]. name=Event_Time token[7]. type=String token[2]. type=Time. Stamp token[8]. name=Action token[2]. format=yyyy/MM/dd HH: mm: ss token[8]. type=String token[3]. name=Source. IP token[9]. name=Message token[3]. type=IPAddress token[9]. type=String token[4]. name=Source. Port 33 © Copyright 2012 Hewlett-Packard Development Company, L. P. token[10]. name=Rule. Number token[4]. type=Integer token[10]. type=Integer The information contained herein is subject to change without notice.

Regex Parser - submessages Common regex - main or common regular expression, for some parsers, this is sufficient Submessage Processing submessageid. token, token that specifies a unique message id submessage. token, token that should be processed as submessage. count, total number of submessages submessage[N]. messageid [if omitted, becomes the default submessage] submessage[N]. pattern. count, how many different patterns a submessage can have submessage[N]. pattern[M]. regex, regex for the pattern submessage[N]. pattern[M]. fields, what fields will be set for the pattern submessage[N]. pattern[M]. mappings [optional, what values will be set to fields] submessage[N]. pattern[M]. extramappings [optional, any extra mappings] 34 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Regex - submesages Subparser regular expressions Suparsers extract additional information from fields created by the common regular expression • • Select the submessage ID Token Select the token to be parsed Create a regular expression to parse the submessage Map the tokens created by the subparser to Arcsight fields submessageid. token=Process. Name submessage. token=Message submessage. count=1 • Submessage ID Value: VPN • Submessage token Value: Encryption failed, username jsmith • Submessage Regex: (. *), username (S+) submessage[0]. messageid=VPN submessage[0]. pattern. count=1 submessage[0]. pattern[0]. regex=(. *), username (\S+) submessage[0]. pattern[0]. fields=event. name, event. destination. User. Name 35 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Regex Parser - submessages 2006: 11: 07 -19: 05: 29 ulogd[1993]: DROP: <1 -10 different drop Messages> 2006: 11: 07 -19: 05: 29 ulogd[1993]: ACCEPT: <1 -10 different drop Messages> 2006: 11: 07 -19: 05: 29 ulogd[1993]: REJECT: <1 -10 different drop Messages> Common Regex will cover 2006: 11: 07 -19: 05: 29 ulogd[1993]: Submessage ID will be DROP – ACCEPT - REJECT 36 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Completed Configuration File # Flex. Connector Regex Configuration File regex=(\d+. \d+): (\S+) (\d+/\d+: \d+) (\d+. \d+): (\d+) \((\S+)\) action=(\S+) Message: \s(. *)? Rule (\d+) token. count=11 token[0]. name=Firewall_IP submessageid. token=Process. Name token[0]. type=IPAddress submessage. token=Message token[1]. name=Process. Name token[1]. type=String event. device. Receipt. Time=Event_Time token[2]. name=Event_Time event. source. Address=Source. IP token[2]. type=Time. Stamp event. device. Address=Firewall_IP token[2]. format=yyyy/MM/dd HH: mm: ss event. destination. Address=Destination. IP token[3]. name=Source. IP event. source. Port=Source. Port token[3]. type=IPAddress event. name=Action token[4]. name=Source. Port event. message=__concatenate(Message, " ", "Rule: ", " ", Rule. Number) token[4]. type=Integer event. destination. Port=Destination. Port token[5]. name=Destination. IP event. device. Severity=Action token[5]. type=IPAddress event. transport. Protocol=Protocol token[6]. name=Destination. Port event. device. Process. Name=Process. Name token[6]. type=Integer token[7]. name=Protocol submessage. count=1 token[7]. type=String submessage[0]. messageid=VPN token[8]. name=Action submessage[0]. pattern. count=1 token[8]. type=String submessage[0]. pattern[0]. regex=(. *), username (\S+) token[9]. name=Message submessage[0]. pattern[0]. fields=event. name, event. destination. User. Name token[9]. type=String 37 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice. token[10]. name=Rule. Number severity. map. high. if. device. Severity=drop token[10]. type=Integer severity. map. low. if. device. Severity=accept

Multiline Regex Configuration parameters identify the start and/or end of each event multiline. starts. regex • A regular expression that identifies when a multi-line event starts. − Example: multiline. starts. regex=|d+/d+ d+: d+|. * multiline. ends. regex (optional) • A regular expression that identifies when a multi-line event ends. − Example: multiline. ends. regex=. *|$ multiline. max. count • Over-flow protection. The Flex. Connector will truncate the message if it reaches the specified number of lines. multiline. delimiter • Lines are concatenated with a space by default. This parameter is used to change the default character. 38 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Extra Processors (parser chaining) Use an extra processor when all or portion of data is suitable for parsing by other parsers Extra processor type map delimited regex keyvalue standardkeyvalue xml ntsubparser 39 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice. Which parser is used Map files Delimited parser Regular expression parser Key value parser with standard delimiters (comma and equals to) XQuery-based XML parser Windows event log parsers

Parser Chaining Example Two or more Flex. Connector types are needed to parse the same data. Example Database Log: 04/06/06 13: 03: 34 1. 2. 2. 2 2435 3. 2. 1. 3 80 Firewall 4. 5. 2. 1 accepted a tcp connection 04/06/06 13: 03: 54 1. 2. 2. 5 53 3. 2. 1. 3 53 Firewall 4. 5. 2. 1 denied a udp connection 04/06/06 13: 04: 23 1. 2. 2. 4 2463 3. 2. 1. 3 22 Firewall 4. 5. 2. 1 denied a tcp connection A regular expression can be used to parse the fifth column. 40 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Conditional Mapping For example, assume the following event: Event id is 532 type A with parameter 3. 3 Event id is 533 type A with parameter root Event id is 534 type A with parameter 3. 3 The regular expression to parse this event is: Event id is (\d+) type (\S+) with parameter (\S+) • You can define three tokens: EVENTID, TYPE, and PARAMETER • For event id 532 or 534, set event. source. Address to 3. 3 • For Event id 533, set event. source. User. Name to root • Without conditional mappings, two regular expressions needed to match—the IP address and the user name • Feasible in this case, but will not scale well 41 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Conditional Mapping Conditional mappings in properties for the above example: regex=Event id is (\d+) type (\S+) with parameter (\S+) token. count=3 token[0]. name=EVENTID token[1]. name=TYPE token[2]. name=PARAMETER #Standard mappings event. device. Event. Class. Id=EVENTID event. device. Event. Category=TYPE conditionalmap. count=1 conditionalmap[0]. field=event. device. Event. Class. Id conditionalmap[0]. mappings. count=2 conditionalmap[0]. mappings[0]. values=532, 534 conditionalmap[0]. mappings[0]. event. source. Address=PARAMETER conditionalmap[0]. mappings[1]. values=533 conditionalmap[0]. mappings[1]. event. source. User. Name=PARAMETER conditionalmap[0]. mappings[2]. event. destination. Address=PARAMETER (DEFAULT) 42 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

WUC (Windows Unified Connector) © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Windows Unified Connector - WUC • Native Java implementation of Windows event collection based on JCIFS • Architecture requires separate parsers for each type of event • Only certain types of Events are supported by WUC oob 44 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Event Collection vs Event Parsing WUC provides capability to COLLECT events from all Windows Event Logs • • Security Event Log System Event Log Common Application Event Log Custom Application Event Logs WUC provides different levels of parsing capabilities • All Security, Core System and Application Events: Complete parsing • Other System and Application Events: Event Header Completely parsed by WUC • Event Body/Description: Use WUC Flex Parser Framework to create flex parsers to parse the Event body 47 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

WUC Flex Parser Framework • Flexible Parser Framework similar to the Arc. Sight Flex Connector Framework • Provides the power and flexibility to create new parsers to parse custom System and Application events Why is it needed? • Security events are generated mostly by the OS • System and Application events are generated by other applications • New applications create new Event Logs (Custom Application Event Logs) • Multiple Event Sources can generate events for the same Event Log How does it parse the System and Application events? • Pre-parses the Event Header fields • Parses the Event Body with a Key Value Parser for each combination of the event’s: − Windows Version − Event Log 48 © Copyright 2012 Hewlett-Packard Development Company, L. P. − The information contained herein is subject to change without notice. Event Source

Windows Event Format • Windows Event = Event Header + Event Description/Body • Event Header format similar for all Windows events • Event Body format differs based on • Windows version • Event Log • Event ID • Event Source 2003 / XP / 2000 System Event Header 2008 / Vista System Event Body Event Header Event Body 49 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

What to Parse? Event Body = Fixed Description String + Variable Place-holders WUC Raw Event: Eventlog. Type=System&&Event. Source=IIS-FTP&&Event. ID=10&&Event. Type=Warning&& User=&&Detect. Time=2008 -12 -10 23: 25: 24&&Event. Category=0&&Computer. Name=ABC &&Key[0]=IEUser@&&Key[1]=192. 168. 21. 251&&Key[2]=120 Windows 2008 Event from Event Log: System, Event Source: IIS-FTP, Event ID: 10 User %1 at host %2 has timed out after %3 seconds of inactivity. 50 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

How to create a WUC Flex Parser? • • Identify the Event Log name, e. g. System, Application, Directory Service, etc… Identify the Event Source, e. g. Service Control Manager Create a Key Value Parser file with the following name format: Format: <Event Log Name>. <Event Source>. sdkkeyvaluefilereader. properties Normalize all the characters in the parser file name Example: system. service_control_manager. sdkkeyvaluefilereader. properties Identify the Windows version of the host Place the parser file in one of the following parser over-ride sub-folder locations Windows Version 51 Parser Over-ride Sub-Folder Location Windows Server 2000 …windowsfgwindows_2000 Windows Server 2003 Windows XP Windows Server 2008 …windowsfgwindows_2003 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice. www. arcight. com …windowsfgwindows_2008

DB-Based Flex. Connector…. . © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

DB Flex Workflow • DBs: Choose ID- or Time-based • Prototype on SQuirre. L first – blame the database! • Transfer through to Flex. Connector 54 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

DB Flex What can go wrong? • Firewall Rules • Credentials (invalid user/pass) • Authorisation (access to table) • Fields exist, and Datatypes • JDBC Drivers (work, and are compatible) • Connection String • Everything else! Use SQuirre. L. Love SQuirre. L. 55 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

j. TDS Why j. TDS? • Open Source Driver supporting MS SQL Server • Supports Windows Authentication! On Con. App! • Apparently faster and more efficient 56 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

j. TDS Driver download http: //jtds. sourceforge. net Driver file goes into /current/lib/agent/jtds_1. 2. 5. jar Database JDBC Classpath (into agent. properties) agents[0]. JDBCDriver=net. sourceforge. jtds. jdbc. Driver Connection String Mixed Mode: jdbc: jtds: sqlserver: //172. 16. 100: 1433; database=SQLServer Windows: jdbc: jtds: sqlserver: //172. 16. 100: 1433; database=SQLServer; domain=WORKGROUP 57 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

syslog © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Syslog Characteristics Do the messages have a syslog header? Syslog Header Facility + Priority Payload Host Name Time Stamp • Is the syslog header rfc compliant? • Syslog-ng with IETF (RFC 5424) is another story • Your parser applies only to the Payload, not to the Syslog Header 59 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Syslog Parsing • Based on a Plugin Framework (subparsers) • Syslog connectors has more 40 subagent parser • When a message is received, it compares the pattern of message to all the available subagent parser till a match is found • First match wins! • Once the subagent parser is matched, it will store this information in syslog. properties of which device events matched which subagent parser • If there is no matched pattern, events will be parsed with the generic parser and store the information in Name field and Device Vendor and Product will be “UNIX” 60 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Other Syslog Flex Connector Considerations If you plan to receive events from multiple devices • You need to write multiple property files, one per device type • Property files are loaded and used in alphabetical order • Regexes defined in properties files need to be unique enough to match only the messages from its device type and not from other device types • Names of the property files should be chosen such that more specific regexes appear in properties files with alphabetically higher names Other tips • If the actual message has data in the delimited, key value or xml format, the regex need only identify the device and an appropriate extra processor can be used for further parsing 61 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Syslog by Example © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Syslog Example Nov 08 15: 40 10. 1. 1. 13 2006: 11: 07 -19: 05: 29 ulogd[1993]: DROP: IN=eth 0 OUT= MAC=00: 0 c: 29: 28: fa: 4 f: 00: 10: 4 b: b 9: 0 e: 84: 08: 00 SRC=10. 1. 1. 2 DST=10. 1. 1. 13 LEN=48 TOS=00 PREC=0 x 00 TTL=128 ID=34476 CE DF PROTO=TCP SPT=2591 DPT=1 SEQ=2195296356 ACK=0 WINDOW=64240 SYN URGP=0 63 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Syslog Example Start Here Nov 08 15: 40 10. 1. 1. 13 2006: 11: 07 -19: 05: 29 ulogd[1993]: DROP: IN=eth 0 OUT= MAC=00: 0 c: 29: 28: fa: 4 f: 00: 10: 4 b: b 9: 0 e: 84: 08: 00 SRC=10. 1. 1. 2 DST=10. 1. 1. 13 LEN=48 TOS=00 PREC=0 x 00 TTL=128 ID=34476 PROTO=TCP SPT=2591 DPT=1 SEQ=2195296356 ACK=0 WINDOW=64240 SYN URGP=0 • • • 64 Syslog subagents are regex parsers This log seems very easy to parse (key/value pairs) but… What about optional fields What about the order of the fields Yes, we can make tokens optional in a regular expression –> inefficient © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Parser chaining (extraprocessors) • Use a regex for: 2006: 11: 07 -19: 05: 29 ulogd[1993]: DROP: • Use a keyvalue parser for the rest of the message IN=eth 0 OUT= MAC=00: 0 c: 29: 28: fa: 4 f: 00: 10 SRC=10. 1. 1. 2 DST=10. 1. 1. 13 LEN=48 TOS=00 PREC=0 x 00 TTL=128 ID=34481 PROTO=TCP SPT=2596 DPT=9 SEQ=2195569882 ACK=0 WINDOW=64240 SYN URGP=0 65 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Keyvalue parser advantages / disadvantages • • • High performance Simple Configuration Arbitrary Key Order Ignores Missing Keys Easier to maintain • Only tool available: notepad/vi • More difficult to debug 66 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Arc. Sight Regex Tool • Start building your base Connector with Arc. Sight Regex tool /bin/arcsight regex • Treat As Syslog • Be as specific as possible • Complete basic mappings • Test with a large sample • Then move on to extraprocessor 67 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Now move on to your favorite editor extraprocessor. count=1 extraprocessor[0]. type=keyvalue extraprocessor[0]. filename=syslog/astaro extraprocessor[0]. field=event. message extraprocessor[0]. clearfieldafterparsing=true extraprocessor[0]. flexagent=true 68 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

astaro. sdkkeyvaluefilereader. properties Token definitions Event mappings token. count=13 token[0]. name=IN token[0]. type=String event. source. Address=SRC token[1]. name=OUT token[1]. type=String token[2]. name=SRC token[2]. type=IPAddress token[3]. name=DST token[3]. type=IPAddress token[4]. name=LEN token[4]. type=String token[5]. name=TOS token[5]. type=String token[X]. name=… 69 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice. event. device. Inbound. Interface=IN event. device. Outbound. Interface=OUT event. transport. Protocol=PROTO event. destination. Address=DST event. destination. Port=DPT event. source. Port=SPT

Test your Connector • Set up a syslog deamon • Feed samples into the connector http: //quignon. de/tools/netsend_jar. zip • Watch the agent. log for ERROR and FATAL [2013 -03 -14 11: 26: 44, 441][FATAL][default. com. arcsight. agent. parsers. k][read. Field. Mappings] Could not load operation [__string. TESTConstant(Rule)] [2013 -03 -14 11: 26: 44, 449][FATAL][default. com. arcsight. agent. parsers. k][read. Field. Mappings] com. arcsight. agent. parsers. operation. Operation. Not. Supported. Exception: Operation [string. TESTConstant] not supported ! at com. arcsight. agent. parsers. operation. Operation. Loader. load. Operation(Operation. Loader. java: 45) at com. arcsight. agent. parsers. operation. Operation. Loader. get. Operation. Index(Operation. Loader. java: 68) at com. arcsight. agent. parsers. j$e_. <init>(j$e_. java: 1283) at com. arcsight. agent. parsers. j. k(j. java: 579) at com. arcsight. agent. parsers. j. d(j. java: 503) at com. arcsight. agent. parsers. j. u(j. java: 347) 70 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Categorization of Flex. Connectors © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

What is Categorization? Categorization allows you to look at an event in a vendor neutral context. Apply Event Categories — six criteria translate the core meaning of an event into Arc. Sight’s Event Schema Object - Entity being targeted Behavior - What is being done to the object Outcome - Result of the Behavior on the object (Success, Failure or Attempt) Technique - Nature of the behavior represented Device Group - Indicates if event is of one type or another (Such as Firewall Events) Significance - Indicates security risk based on various data points, information from the device, and the data model 72 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Categorisation • If you have time, or is a POC requirement • Beware of existing content triggering on this (eg. Config Changes, Brute Forces) 73 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice. https: //irock. arcsight. com/message/3915 https: //irock. arcsight. com/docs/DOC-2290 ESM 4. 0 User Reference Guide / Data Fields ESM 5. 0 ESM_User. Guide. pdf

Categorization files are map files Place categorization files in <connector_home>/user/agent/acp/categorizer/current/<vendor_name>/ Create a file product. csv which will reside inside of the directory Vendor and product must match what is being assigned in the connector configuration file 74 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Categorization File Product. csv File should contain a header like this event. device. Event. Class. Id, set. event. category. Object, set. event. category. Behavior, set. event. category. Technique, set. event. category. Device. Group, set. event. category. Significance, set. event. category. Outcome The key field is device. Event. Class. Id this is how we assign the categorization to the events 75 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Documentation • Flexcon Dev Guide sucks but is a good starting point • Review PS / Support Training recordings • Confluence! 76 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

The Burden With The Escape Characters • • • 77 In literal Java strings the backslash is an escape character The literal string "\" is a single backslash In regular expressions, the backslash is also an escape character The regular expression \ matches a single backslash This regular expression as a Java string, becomes "\\" That's right: 4 backslashes to match a single one © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Some Useful Undocumented Token Operations __divide(integer, integer) __sum(integer, integer [, integer. . . ]) __substract(. . . __product(. . . Curious for more? –Unzip the arcsight-parserframework<version>. jar –Look into comarcsightagentparsersoperation 78 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Event merging Event Merger Some devices will send information about a single event in multiple log lines Even though in some cases it would be fine to send each line as a single event, in some other instances it is necessary to merge the information of all the events into a single one. Event Merger vs. Multi-line regex One could argue that a multi-line regex agent could be developed for cases where multiple events have to be merged into a single one In some instances the events sent by the device will not necesarilly be close together, there could be events that will be sent in between other events 79 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Event merging Consider the following log lines: [18/Jul/2005: 12: 30: 20 -0400] conn=8 op=0 msg. Id=82 - BIND uid=admin [18/Jul/2005: 12: 30: 25 -0400] conn=7 op=-1 msg. Id=-1 - LDAP connection from 10. 0. 20. 122 to 10. 0. 20. 122 [18/Jul/2005: 12: 30 -0400] conn=8 op=0 msg. Id=82 - RESULT err=0 • Two of those lines refer to a "binding" operation where the user id is admin and the error is 0. This example could NOT be solved using a multiline. • Both events have a connection (conn and a message id (msg. Id) that is identical for both events • Search Confluence for „Event Merger“ 80 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Event Name Event names should not have variables %PIX-3 -307001: Denied Telnet login session from 1. 1 on interface inside The Good Denied Telnet The Bad Denied Telnet login session from 1. 1 th. E Ug. Ly %PIX-3 -307001: Denied Telnet login session from 1. 1 on interface inside 81 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Arc. Sight Regex Tool • • 82 Always use the version of the target connector (latest if possible) Delete tmp and bak files after using it Stop using it once you have modified properties file outside of the Tool Rememer to double-escape when editing with external editor (s+) turns into (\s+) © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Regular Expressions Regular expressions should be as specific as possible abc, def, ghi The Good \S+? , \S+? or even better [^, ]+, [^, ]+ The Bad \S+, \S+ th. E Ug. Ly. *, . * 83 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Regular expressions for Syslog sub-agents Generic regular expressions may cause legitimate Unix syslog messages to not be detected abc, def, ghi The Good (Order is from most specific to most generic) abc, def, ghi (exact match) [^, ]+, [^, ]+ \S+? , \S+? . *? , . * The Bad/th. E Ug. Ly (Order is from most generic to most specific). *? , . * (generic catch-all) \S+? , \S+? [^, ]+, [^, ]+ abc, deg, ghi 84 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Syslog subagent Flex: Other parser matches first If possible, remove other subagents from framework • Set usecustomsubagentlist to true • Remove parser from customsubagentlist If you still need the other parser • Change parser order 85 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Fields to map • • 86 source. Address destination. Address device. Receipt. Time device. Event. Class. Id device. Vendor Categorization depends on these! device. Product name © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Custom Fields • Each Device Custom field has a corresponding Device Custom label, it is a good practice to make sure the labels are also set 87 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Device Receipt Time, Start Time, End Time Device Receipt Time: What is this? Time when the device “detected” the event Should this be set? YES! This MUST be set in the parser Start Time: What is this? Time when the actual event detection started Should this be set? Only when device calls it out explicitly End Time: What is this? Time when the actual event detection ended or when the actual event ended. Should this be set? Only when device calls it out explicitly 88 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Sidetables • Only relevant to ESM/Express based on Oracle • Certain fields with highly repetitive content are cached in sidetables • Mapping wrong information into those fields will cause sidetable overflow • Wrong field mapping can kill or slow-down ESM (pre-CORRE) 89 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Fixing existing parsers (Smart. Connectors) © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Parser overrides • property file contains the modified information ONLY to replace the corresponding line(s) in the original parser file, in order to change the data mapping • Must have the same parser file name as the original parser • Placed under currentuseragentfcp<folder_ name><Parser_file_name> • The <folder_name> must be same as the original parser file’s folder name 91 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Unobfuscated property file # Unobfuscated properties file # Copyright 2001 -2005 Arc. Sight, Inc. All Rights Reserved. # This software is the proprietary information of Arc. Sight, Inc. # Use is subject to license terms. # IMPORTANT: Delete the following property when you modify this # file! ignore. this. file=x # Delete the following property if this file contains only # overrides, or leave it as is if this file is intended to be # complete replace. defaults=x #: : : : : : : : : : : : # Apache sdk regex properties file: For Apache access and error logs #: : : : : : : : : : : : regex=(apache_access_log|apache_error_log): \s*(? : \[ID (\S+? )\])? \s*(. *) token. count=5 92 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Parser overrides Only the changes need to go into the prop file when you remove: “replace. defaults=x” regex=(apache_access_log|apache_error_log): \s*(? : \[ID (\S+? )\])? \s*(. *) 93 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

How to get the original parsers • Ask a friend in support • Unobfuscate yourself • Start Connector on commandline: . /arcsight agents –unobpswd <code> • Parsers will end up in folder: user/agent/aup/fcp • Code changes monthly and has to match either • • Month when Connector build was produced Month of AUP version deployed on top of that Connector • Need a code? Ask a colleague from the Arc. Sight Specialists team 94 © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.

Thank you © Copyright 2012 Hewlett-Packard Development Company, L. P. The information contained herein is subject to change without notice.