Best Practices in I T Logging and Log

Best Practices in I. T. Logging and Log Analysis February 17 th, 2004 Mark Lachniet, Analysts International

Introductions • Mark Lachniet, Technical Director of Analyst International’s Security Services Group • Technical lead developing for services, methodology, quality control, technical presales • Certified Information Systems Auditor (CISA) from ISACA • Certified Information Systems Security Professional (CISSP) ISC^2 • Linux LPIC-1, Novell Master CNE, Microsoft MCSE, Checkpoint CCSE, Tru. Secure ICSA, etc. • Former I. T. director of Holt Public Schools • Frequent speaker for local organizations

Agenda • • • The forensic approach to logging The Syslog protocol UNIX logging Windows logging Internet history logging Network device logging Policies and procedures Resources Q&A / Discussion

Why Logging? • http: //www. cerias. purdue. edu/homes/rgk/at. html – Individual Accountability – Reconstructing Events – Problem Monitoring – Intrusion Detection • The presence of a good logging system can also be a deterrent • Logging may be mandatory for certain laws and regulations (Sarbannes-Oxley, GLBA, etc. ) • Logging should be required through organizational policy!

The Forensic Approach to Logging • As a security analyst, I analyze a lot of intrusions after the fact, less so fraud • Almost always, there is very limited information to use in recreating the nature of a compromise • In an environment with good internal controls, it may be possible to find a great deal of information, on many disparate systems • The goal here is to be able to recreate the sequence of events that lead to an incident • Due to the fact that many incidents occur in ways that don’t leave a trail (e. g. a security flaw in an application) we must cover as many bases as possible

Common Sources of Logging Data

Log Data and Formats - Syslog • There a lot of places where log data can be gathered, and they don’t all use the same protocol • Historically, the most commonly used format for logging is Syslog (from the UNIX world) • Syslog works as a network-aware component, with both a server and client programs • The server may be UNIX syslogd, Kiwi, or a similar “listener” that logs data or takes action • Syslog servers typically listen on UDP port 514, and transmit data in an unencrypted format • Connections may be made over a physical network, or via a local loopback address • Log entries are analyzed and processed

Log Data and Formats – Syslog Cont. • Based on its configuration, actions may be taken: – Write to file (one or more) – Send to another syslog server – Perform an action (send an e-mail or page, play a sound file, log to a database, etc. ) • Syslog uses two criteria for its alerts – facility and level – usually denoted in a format such as: – mail. info (routine mail entry - log to mail log) – kern. crit (kernel emergency! Email admin) • These criteria aren’t really implemented in a standard way across operating systems, but you can use them for your own purposes • Note: Old syslog has poor security – you need another layer of security such as hostname / IP filtering, or use a better product such as Syslog NG.

Example /etc/syslog. conf # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern. * /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *. info; mail. none; authpriv. none; cron. none /var/log/messages # The authpriv file has restricted access. authpriv. * /var/log/secure # Log all the mail messages in one place. mail. * /var/log/maillog

Example Syslog File Security Info System Problem Feb 16 09: 26: 32 lachniet dhclient: DHCPREQUEST on eth 0 to 12. 242. 20. 34 port 67 Feb 16 09: 26: 32 lachniet dhclient: DHCPACK from 12. 242. 20. 34 Feb 16 09: 26: 32 lachniet dhclient: bound to 67. 162. 209. 240 -- renewal in 2969 se conds. Feb 16 09: 57: 07 lachniet sshd(pam_unix)[19088]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=host 34. lan. sequoianet. com user=root Feb 16 09: 57: 12 lachniet sshd(pam_unix)[19088]: session opened for user root by (uid=0) Feb 16 09: 57: 50 lachniet sendmail: sendmail shutdown succeeded Feb 16 09: 57: 50 lachniet sendmail: sm-client shutdown succeeded Feb 16 09: 57: 51 lachniet sendmail: Warning: Option: Auth. Options requires SASL su pport (-DSASL) Feb 16 09: 57: 51 lachniet sendmail: sendmail startup succeeded Feb 16 09: 57: 51 lachniet sendmail: sm-client startup succeeded

More on Syslog • Syslogging only happens if there is a client program to send the message • This is usually built into the application itself, such as the kernel, or a sendmail program • Many programs generate log data, but do not have the coding to send it to the syslog facility, or don’t do it because its impractical (e. g. a web server) • To promote enterprise security and efficiency, these programs can be reprogrammed, or can call a supplemental program such as ‘logger’ (included in most UNIX operating systems) # logger -p kern. crit Your computer is on fire! Creates a log entry such as: Feb 16 14: 33: 29 lachniet root: Your computer is on fire!

Microsoft Windows Logging • Microsoft products to *not* use the Syslog protocol for logging by default • Instead, there a number of ways to log data, and disparate places to log it: – Windows Event Log (MMC plugin) – Internet Information Systems Logs (www, FTP, SMTP) – DHCP server logs – Microsoft Exchange – DNS logs, etc. • Because of this, it is difficult to set up centralized logging for Windows machines “out of the box” (note: UNIX has the same problem) • Fortunately, a number of products have been created to help us with this task

Windows Event Log Example The Event ID is a critical component that you can search for Supplemental information can also be useful

Windows Event Log Example Cont. • The previous was a perfect example of something you might want to know about • Not even this level of information is available by default – it must be manually configured on most systems (Windows 2003 may be an exception) • !! The default level of logging is useless!! • As an auditor, no system should be “green lighted” unless there is some kind of formal system hardening methodology that includes adequate logging of the operating system • Also, note that the event log only tracks some events, and does not include web servers, etc.

Configuring Logging on Windows • In order to get useful logging out of a Windows product, you have to manually turn this on • If you are configuring for an individual workstation, you are going to configure a ‘local security policy’ – this can be done by a machine administrator (usually an end user) • If you are doing this for a large group of people, presumably in Windows 2000 Active Directory, you will use Group Policy • In any case, you need to manually configure what your policy settings will be – usually through the “Administrative Tools” or mmc program • Lets look at an example of a local workstation – doing this for a domain is similar but more complicated (there may be more complex inheritance rules)

Default Windows Event Logging By default, nothing is audited

Enabling Auditing • Double-click on the category, and select the check boxes for important items

Configuring Logging on Windows Cont Now you are auditing logons and logoffs • Repeat as desired! (use a guide to determine what settings to use)

Configure Event Viewer Settings • But wait, you are not done – the default “event viewer” settings may not be adequate • Within the event viewer, you need to set: – The size of the event log – What happens when the event log reaches maximum size (overwrite, rollover, nothing) – How many days to store the logs for before rolling over • There is also the option to “hard stop” the server if it cannot log – this may be needed for high-security systems. See: http: //www. microsoft. com/technet/treeview/default. asp ? url=/technet/columns/security/askus/aus 1101. asp

Setting Event Log Properties

Setting Event Log Properties

Windows Event Logging • For a list of Windows event ID’s check out: http: //support. microsoft. com/? kbid=299475 http: //support. microsoft. com/? kbid=301677 • There a variety of things that you can audit for: – – – Login / logout Account management (create, delete, modify) Directory service access Object access (files, printers) Use of privileges (backup) • Yet, there are many things you can’t monitor with event log: – Internet Information Server – Microsoft Exchange, etc.

The Honey Pot • If there is a particularly interesting file on your network, for example “salary. xls” or “resignation. doc”, you may want to audit all access to this file • You may even choose to create this file just to see who looks at it • In another example, you may wish to audit access to log files, and filter out normal users – most folks have no business reading log files • This can take place on file servers (network shares) but can also be configured on your local workstation

Good Windows Log Features • The ability to forward Windows events to a centralized syslog server for aggregation and analysis • The ability to log events to a SQL database for complicated analysis and trending • The ability to maintain time synchronization with other systems (NTP) • The ability to recognize malicious use (IDS) • Regular batch hashes, zips and transfers of log data to repository or read-only medium (CD-R? ) • The ability to ‘hash’ a file (i. e. , create a cryptographically sound representation of the file) may be required for high-security

Other log sources on Windows • File MAC (M)odify (A)ccess (C)reate times are better than nothing, but can be circumvented • The Windows registry has a time/date stamp of the last access • Local workstations’ Internet history is usually very interesting to analyze • Even if the Internet cache has been cleared, there may be remnants that can be undeleted • Additionally, the ‘index. dat’ file is a repository of browsing history • A good product to analyze index. dat files is at: http: //digitaldetective. co. uk • As an auditor, whatever you can do to promote the security of these log sources is a good thing, but there may not be much you can do.

Application and Database Logs • There are many applications aside from the operating system, and they almost all have their own logging system • Obviously the level of logging required will depend upon the criticality of the application • Databases have varying levels of logging capability, but beware the system overhead! (it may take more cycles to log the activity than to actually perform the database request) • Due to the potential difficulty of logging on the database server side, implement “defense in depth” and do extensive logging from the application side • As an auditor, the logging and auditing capabilities of any business-critical application should be evaluated (preferably before it is purchased)

Logging From Web Applications • Defeating web application logic is the new frontier in hacking, and most logging is for the web server and operating system only, and not that useful • A recent study found that at least 92% of web applications had vulnerabilities (http: //www. vnunet. com/News/1152521) • My personal experience finds that this is about right! If you have Internet-facing applications, you definitely need to look at them • In UNIX, it is possible to call ‘logger’ or similar logging facilities through CGI binaries • In windows, there are pieces of code you can recycle to do syslog, including some from: http: //www. kiwisyslog. com/products. htm#logger

Logging on the Network

Logging Sources on the Network • Due to the fact that many attacks or fraud cases don’t leave a good audit trail on the host or server, network logging be a great help • Devices that could / should trap useful data – – – Routers Firewalls Intrusion Detection Systems Content filters (e-mail, web) Proxy servers (squid, ISA server) • As with hosts, proper configuration is essential – many firewalls do not trap full “packet level” information (i. e. detailed TCP/IP headers) • In addition, a good network design is required so that an intermediary device such as a firewall can even see the traffic

Network Logs • Network device logs can provide a detail of what type of information traveled between network systems: – Determine how the system was profiled (reconnaissance) – Determine how the system was attacked (vulnerability) – Determine what happened after the attack – did the hacker use your system to store files? Attack other systems? Copy data to another machine? – Determine if multiple parties were involved (may indicate collusion on the part of multiple employees)

Log Aggregation and Analysis • Logging is important, but making sense of the data is even more important • If individuals manually analyze large datasets, they may get desensitized and miss something important • Additionally, manually viewing single events (a packet, a single URL) does not really establish a useful baseline • Is this event part of an attack? Is it some indication of a performance problem? • For this reason, a variety of log analysis tools have been created to help the end user to deal with the deluge of data

Job Roles and Responsibilities • The work tasks of log analysis should be explicitly defined, preferably as part of a job description, and employees should be evaluated on this • Log analysis procedures should be documented, and a log of log analysis kept • The criteria for what is an “interesting” event should be defined to the greatest degree possible • The ability to analyze the logs should be held by more than one individual, both as a quality check, and as security for the organization • An occasional external survey of how log analysis is performed might be of benefit – they may know of other tools or events to be aware of • All logs should be backed up in multiple places and kept as long as needed (a criteria that should be defined by management, particularly in SEC regulated industries, or those subject to FOIA)

Snare • Snare - Audit and Event. Log analysis and forwarding • http: //www. intersectalliance. com/

Kiwi Syslog Daemon • Receives the syslog events from local programs and remote systems (as needed)

Kiwi Syslog Daemon Cont. • Kiwi can do lots of neat things, like send you a page or play a WAV file every time there is a failed administrator login

e. IQ System Analytics • A “Web-based analysis and reporting solution that can analyze event logs generated by Windows and UNIX networks”

Net. IQ (Web Trends) Firewall Suite • Converts large amounts of firewall data into handy charts and graphs • Tracks both “packet” and “web browsing” data

On Selecting Products • Note – I do not have a strong preference for any one tool and do not recommend any of the above • Much of what needs to happen can be done entirely on open source and freeware tools • The biggest cost is labor. Do you spend the money on a product or a person? What is the accuracy? • As with anything, the capability to log and audit usage should be a component of the System Development Life Cycle and not in isolation • Consult with subject matter experts where appropriate – technical people know the capabilities, but not the needs or the reasoning for this level of detail (and work) in the organization

Links http: //www. doit. wisc. edu/security/resources/bestpract/logging. asp (windows settings) http: //www. sans. org/rr/papers/index. php? id=1100 (using Kiwi in a Cisco and Microsoft environment) http: //www. sans. org/rr/papers/index. php? id=1101 (Syslog in a Microsoft environment) http: //www. owasp. org (best practices in web application development) http: //www. databasejournal. com/features/mssql/article. php/2243271 (Enhanced SQL Security auditing) http: //www. winsyslog. com/en/FAQ/index. php (syslog products for Windows) http: //www. campin. net/syslog-ng/faq. html (Syslog NG products) http: //www. loganalysis. org/ (web site / listserve dedicated to log analysis) http: //www. loganalysis. org/sections/audit/tbird-syslog-overview. pdf (overview of Syslog in general)

More Links http: //windows. about. com/library/tips/bltip 316. htm (turning on logging of DHCP) http: //www. microsoft. com/technet/treeview/default. asp? url=/technet/colu mns/security/askus/aus 1101. asp (enabling auditing on Windows) http: //www. jsiinc. com/SUBI/tip 4100/rh 4108. htm (list of security events to audit for) http: //www. eiqnetworks. com/products/systemanalytics. shtml (commercial product to analyze Windows and UNIX logs) http: //www. mwagent. com/en/ (Monitor. Ware) http: //www. sans. org/rr/papers/index. php? id=201 (Effective logging and the use of the Kiwi syslog utility)

Discussion • This presentation to be available at: http: //lachniet. com/powerpoint Mark Lachniet CISSP, CISA, MCSE, MCNE, CCSE, LPIC-1, TICSA Technical Director, Security Group Analysts International (517) 336 -1004 (voice) (517) 336 -1100 (fax) mailto: mlachniet@analysts. com