Using OSSEC Open Source Host Based Intrusion Detection

Using OSSEC Open Source Host Based Intrusion Detection Justin C. Klein Keane University of Pennsylvania School of Arts & Sciences Information Security and Unix Systems ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

Presentation Format Top down High level definition Capabilities Customization Our Implementation Logistics ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

Network vs. App Security • We've won network security • Made it more expensive to compromise networks than applications • Defending applications requires new approaches • OSSEC fits easily and naturally into an application security program ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

About OSSEC Open source host based intrusion detection system (HIDS) Written by Daniel Cid, who continues to participate Pronunciation varies Acquired by Third Brigade in 2008, which was then acquired by Trend Micro in 2009 ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

What is an IDS Intrusion detection system (IDS) Distinct from an intrusion prevention system (IPS) Monitor data points to identify malicious behaviour Anomaly detection or pattern matching Basically just automated traffic monitoring ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

Differences of HIDS vs. NIDS HIDS – host based IDS NIDS – network based IDS NIDS uses data from the wire HIDS can use filesystem as well as network – – Log files, file contents Filesystem attributes Main difference: visibility ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

Advantages of HIDS Low(er) false positive rate If your HIDS detected the traffic it was definitely parsed by the target Leverages existing hardware Not very resource intensive ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

Disadvantages of HIDS Blind spots: If it isn't logged, or stored on the filesystem, it's invisible Unable to parse unknown or unlogged traffic Can't evaluate egress Can't spot probes that don't hit active services Requires an agent install on endpoints Can be defeated with a host compromise ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

Advantages of NIDS All traffic becomes visible, ingress and egress Requires just one point of installation Failure doesn't affect endpoints ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

Disadvantages of NIDS High false positive rate Easy to overwhelm with bad traffic or evade – No way to tell if attack was “real” Packet fragmentation, etc. Can't see encrypted traffic ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

OSSEC Beyond HIDS Filesystem integrity checking Registry monitoring on Windows Active response Commonly used for RTBL Can be scripted for almost any behaviour Rootkit detection ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

Why Penn is Using OSSEC • Found OSSEC in a search for open source SEM and SIM solutions • No access to network taps or span ports • Looking for a sensor net to correlate alerts • Using OSSEC for over 3 years now ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

How OSSEC Works Three modes Local, client, server Client server model Clients receive configuration from server Clients send logs to server over an encrypted channel ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

OSSEC Network Communication • UDP port 1514 • Compressed messages (using zlib) • Blowfish based encryption using pre shared keys • Logs specified on client are collected and sent to manager for analysis ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

OSSEC Data Flow Graphic concept courtesy http: //ossec. net/ossec-docs/auscert-2007 -dcid. pdf ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

OSSEC Decoders • OSSEC uses decoders to parse log files • Decoders are written as XML • Extracts useful data fields from log entries to use for rule and alert matching including: – – Source IP and/or port Destination IP and/or port Program name or user name And more. . . ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

OSSEC Rules • OSSEC rules are stored as XML files • Rules are hierarchical • By default OSSEC includes rules for: apache, arpwatch, asterisk, cisco ios, courier, firewalls, ftpd, horde/imp, IDS systems, IMAP, Mc. Afee antivirus, MS auth, MS DHCP, Exchange, Microsoft FTPD, My. SQL, Bind, Netscreen, PAM, postfix, Postgres, Pro. FTP, Roundcube, sendmail, samba, Squit, SSH, Symantec AV, Syslog, Telnet, VMWare, VSFTP, Wordpress, and more. . . ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

Rule Format Rules are assigned priority levels, usually from 1 (lowest) to 15 Rules trigger based on: Pattern matching in strings Timing between matches Dependence on other rules Time of day Hostnames Applications ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

OSSEC Alerts • Default settings include alerting on: – – – • Web attacks SSH brute force Buffer overflows and program crashes Firewall events Users using sudo Many more. . . Creating custom alerts is relatively easy ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

Alert Behavior • When a rule triggers an alert several actions can be configured: – – Logging Sending an e mail alert • – Sending a SMS alert Executing an active response script ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

Active Response Initiates scripts Default installations include firewall rules Time based Whitelists prevent self denial of service Active response can be delegated to multiple hosts ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

Custom Decoders and Logs • OSSEC will monitor many common services out of the box • OSSEC can be extended to monitor custom applications • OSSEC can detect events in custom application logs and trigger custom active response scripts ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

Log File Monitoring • OSSEC monitors specific logs by default, including: – – – • Syslog Apache http logs Mail logs OSSEC can be configured to monitor any log it can gain access to ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

Beyond HIDS: Monitoring Scripts • OSSEC can be used to monitor the output of custom scripts • For instance, OSSEC can generate alerts based on changes to NMAP scan results of specific hosts – • http: //www. ossec. net/wiki/Know_How: Nmap Can also log scripted alerts to common log (syslog) and write a rule to augment existing log monitoring ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

Generating Reports OSSEC includes ossec reportd Can be used to generate summary reports Ex. show all brute force attempts and usernames used and number of times attempted OSSEC can also log to a database so that SQL can be used for reporting Custom scripts can be used to parse alert logs Web based user interface (WUI) reporting ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

OSSEC Community Extremely active user community Developer mailing list OSSEC mailing list (and Google group) http: //groups. google. com/group/ossec list OSSEC wiki http: //www. ossec. net/wiki Commercial support from Trend Micro ossec. purchase@trendmicro. com ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

Solutions • OSSEC is a service you can utilize internally or offer your stakeholders • OSSEC allows you to extend your security impact • OSSEC provide you greater visibility into your security posture ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

How Penn is Using OSSEC • Server installed on a bastion host • Agents deployed on critical hosts • Separate server installed on cluster back ends • Alerts go to queue monitored by security staff • Per machine alerts go to individual sysadmins • Using OSSEC to monitor custom alert scripts ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

Logistical Considerations • As with any intrusion detection system, expect a timeframe for tuning • Be extremely careful with active response to avoid self denial of service • Installation may require coordination with sysadmins ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

Overall Impact • Develop metrics to justify security resource allocation • Verify effectiveness of security countermeasures • Track outlying machines, especially virtual machines • Extend service offerings to client groups • Centralized log reporting ©Justin C. Klein Keane <jukeane@sas. upenn. edu>

Pitfalls of OSSEC • Difficulty in upgrades between versions • Coordinating pre shared keys can be problematic • Sometimes agents become unresponsive • Volume of alerts • In testing OSSEC is great for early warning but not so good in a post compromise situation ©Justin C. Klein Keane <jukeane@sas. upenn. edu>