Unity and Disunity of Unix Log File Management
- Slides: 44
Unity and Disunity of Unix Log File Management Tools Dusan Baljevic Sydney, Australia © 2008 Dusan Baljevic The information contained herein is subject to change without notice
Unix Log Files and Their Management Tools - Present • Most of the time, admins rely on Shell or Perl scripts • As well, find command is commonly used for cleanups 03 November 2020 Webinar - Dusan Baljevic 2
Unix Log Files and Their Management Tools Solaris logadm Linux logrotate AIX (built-in log file rotation and compression) HP-UX (various tools) 03 November 2020 Webinar - Dusan Baljevic 3
AIX syslog • AIX has built-in log file rotation and compression. They are optional fields • Format msg_src_list destination [rotate [sizek|m] [files] [timeh|d|w|m|y] [compress] [archive]] msg_src_list is a semicolon separated list of facility. priority facility all (except mark) mark - time marks kern, user, mail, daemon, auth, . . . priority is one of (from high to low): emerg/panic, alert, crit, err(or), warn(ing), notice, info, debug (meaning all messages of this priority or higher) destination is: /filename - log to this file username[, username 2. . . ] - write to user(s) @hostname - send to syslogd on this machine * - send to all logged in users 03 November 2020 Webinar - Dusan Baljevic 4
AIX syslog Features (part 1/2) • If destination is a regular file and the word rotate is specified, then the destination is limited by either size or time, or both. The backup filenames are created by appending a period and a number to destination, starting with. 0. The time value causes the destination to be rotated after time. If both time and size are specified, then logfiles will be rotated once the logfile size exceeds size or the after time, whichever is earlier • If the compress option is specified then the logfile names will be generated with a. Z extension. The files keyword will be applicable to the logfiles which are currently under rotation. For example, if we specify the compress option, then only file with. Z extension will be under rotation and the number of such files will be limited byfiles. Any logfiles with an extension other than. Z will not be under the rotation scheme and thus will not be under the restriction of files. Similarly if the compress option is removed then the files which have been generated with. Z extension will no longer be the part of rotation scheme and will not be limited by the files 03 November 2020 Webinar - Dusan Baljevic 5
AIX syslog Features (part 2/2) • The minimum size that can be specified is 10 k. The minimum number of files that can be specified is 2. The default size is 1 MB and the default for files is unlimited. Therefore, if only rotate is specified, the log will be rotated with size = 1 m. The compress option means that rotated log files that are not in use will be compressed. The archive option will save rotated log files that are not in use to archive. The default is not to rotate log files • The letter indicating the unit must immediately follow the number in the syntax. For example, to specify the log rotation of every two days, the phrase time “ 2 d” is correct, but “ 2 d” is not 03 November 2020 Webinar - Dusan Baljevic 6
AIX /etc/syslog. conf • Example /etc/syslog. conf *. info /var/adm/syslog. log *. alert /var/adm/syslog. log *. notice /var/adm/syslog. log *. warning /var/adm/syslog. log *. err /var/adm/syslog. log *. crit /var/adm/syslog. log rotate time 1 d files 9 daemon. debug /var/adm/ftpd. log rotate size 1024 k files 5 03 November 2020 Webinar - Dusan Baljevic 7
Linux logrotate • It allows automatic rotation, compression, removal, and mailing of log files. Each log file may be handled daily, weekly, monthly, or when it grows too large. Normally, logrotate is run as a daily cron job. It will not modify a log multiple times in one day unless the formula for that log is based on the logs size and logrotate is being run multiple times each day, or unless the “-f” or “-force” option is used. • Cron job /etc/cron. daily/logrotate #!/bin/sh /usr/sbin/logrotate /etc/logrotate. conf EXITVALUE=$? if [ $EXITVALUE != 0 ]; then /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]" fi 03 November 2020 Webinar - Dusan Baljevic exit 0 8
Linux /etc/logrotate. conf weekly rotate 4 create dateext include /etc/logrotate. d /var/log/wtmp { monthly create 0664 root utmp rotate 1 } /var/log/btmp { missingok monthly create 0600 root utmp rotate 1 } 03 November 2020 Webinar - Dusan Baljevic 9
Linux logrotate Command Usage: logrotate [OPTION. . . ] <configfile> -d, --debug Don't do anything, just test (implies -v) -f, --force Force file rotation -m, --mail=command Command to send mail (instead of `/bin/mail') -s, --state=statefile Path of state file -v, --verbose Display messages during rotation Help options: -? , --help Show this help message --usage Display brief usage message 03 November 2020 Webinar - Dusan Baljevic 10
Linux /etc/logrotate. d Directory # ls /etc/logrotate. d bittorrent fail 2 ban munin-node rpm squid yum clamav-update httpd nagios samba squid. Guard zabbix collectl mgetty named sa-update syslog cups mimedefang ppp setroubleshoot tux dirmngr munin psacct snmpd wpa_supplicant 03 November 2020 Webinar - Dusan Baljevic 11
Linux /etc/logrotate. d Example # cat /etc/logrotate. d/httpd /var/log/httpd/*log { missingok notifempty sharedscripts postrotate kill -HUP `cat /var/run/httpd. pid 2>/dev/null` 2> /dev/null || true endscript } 03 November 2020 Webinar - Dusan Baljevic 12
Solaris logadm • Starting from Solaris 9, there is a standard tool, called logadm, to rotate logs • logadm is an independent utility (unlike integrated in syslog daemon capability that can be found in AIX). The logadm command is a preconfigured entry in the default crontab file supplied starting with Solaris 9 • /usr/lib/newsyslog script is no longer used • Before Solaris 9 there was Free. BSD-style tool newsyslog located in /usr/lib and Perl script rotatelog. It was run from cron. For Solaris 8 and earlier download the tar. gz file, untar it, go in the new directory, and execute make install. It will install /usr/local/sbin/rotatelog and /usr/local/etc/rotatelog. conf 03 November 2020 Webinar - Dusan Baljevic 13
Solaris /etc/logadm. conf (part 1/2) logadm. conf specifies the schedule for log rotation and options with which rotation will be performed. The default configuration: /var/log/syslog -C 8 -P 'Sun Sep 14 17: 10: 00 2008' -a 'kill HUP `cat /var/run/syslog. pid`‘ /var/adm/messages -C 4 -P 'Fri Sep 12 17: 10: 00 2008' -a 'kill -HUP `cat /var/run/syslog. pid`' /var/cron/log -P 'Fri Aug 22 17: 10: 00 2008' -c -s 512 k -t /var/cron/olog /var/lp/logs/lpsched -C 2 -N -t '$file. $N' 03 November 2020 Webinar - Dusan Baljevic 14
Solaris /etc/logadm. conf (part 2/2) /var/fm/fmd/errlog -M '/usr/sbin/fmadm -q rotate errlog && mv /var/fm/fmd/errlog. 0 - $nfile' -N -s 2 m smf_logs -C 8 -s 1 m /var/svc/log/*. log /var/adm/pacct -C 0 -N -a '/usr/lib/accton pacct' -g adm m 664 -o adm -p never /var/log/poold -N -a 'pkill -HUP poold; true' -s 512 k /var/fm/fmd/fltlog -A 6 m -M '/usr/sbin/fmadm -q rotate fltlog && mv /var/fm/fmd/fltlog. 0 - $nfile' -N -s 10 m 03 November 2020 Webinar - Dusan Baljevic 15
Solaris logadm Command Usage (part 1/3) Usage: logadm [options] (processes all entries in /etc/logadm. conf or conffile given by -f) or: logadm [options] logname. . . (processes the given lognames) General options: -e mailaddr mail errors to given address -f conffile use conffile instead of /etc/logadm. conf -h display help -N not an error if log file nonexistent -n show actions, don't perform them -r remove logname entry from conffile -V ensure conffile entries exist, correct -v print info about actions happening -w entryname write entry to config file 03 November 2020 Webinar - Dusan Baljevic 16
Solaris logadm Command Usage (part 2/3) Options which control when a logfile is rotated: (default is: -s 1 b -p 1 w if no -s or -p) -p period only rotate if period passed since last rotate -P timestamp used to store rotation date in conffile -s size only rotate if given size or greater Options which control how a logfile is rotated: (default is: -t '$file. $n', owner/group/mode taken from log file) -a cmd execute cmd after taking actions -b cmd execute cmd before taking actions -c copy & truncate logfile, don't rename -g group new empty log file group -l rotate log file with local time rather than UTC -m mode new empty log file mode -M cmd execute cmd to rotate the log file -o owner new empty log file owner -R cmd run cmd on file after rotate -t template for naming old logs -z count gzip old logs except most recent count 03 November 2020 Webinar - Dusan Baljevic 17
Solaris logadm Command Usage (part 3/3) Options which control the expiration of old logfiles: (default is: -C 10 if no -A, -C, or -S) -A age expire logs older than age -C count expire old logs until count remain -E cmd run cmd on file to expire -S size expire until space used is below size -T pattern for finding old logs 03 November 2020 Webinar - Dusan Baljevic 18
Solaris logadm and Timezone • By default, logadm works in GMT. All entries written to the /etc/logadm. conf file will have a GMT timestamp • Use the “-l” option to set logadm to local time 03 November 2020 Webinar - Dusan Baljevic 19
Solaris logadm – Example for wtmpx • Add into /etc/logadm. conf /var/adm/utmpx -C 12 -P ‘Mon Oct 13 17: 00 2008' -s 100 m -z 0 "-C 12" means it will preserve 12 versions of the log file "-P. . " means when to first start processing the log file "-s 100 m" defines the maximum size of the log file before it is rotated "-z 0" sets the gzip compression • Run command: # logadm • Check it: # logadm -V 03 November 2020 Webinar - Dusan Baljevic 20
HP-UX 11 i syslogd • HP-UX 11 i v 1 and earlier do not have log file automation • HP-UX 11. 23 and later syslogd logs messages into a set of files. Once the size of a log file reaches 2 GB, syslogd stops logging to that file. Configure the maximum size of syslogd log files by setting the variable LOG_SIZE in /etc/default/syslogd The value of LOG_SIZE can be any positive integer greater than 2, representing the maximum size of the file in GB. When LOG_SIZE=NOLIMIT, syslogd uses the limit imposed by the file system on file size 03 November 2020 Webinar - Dusan Baljevic 21
HP-UX 11 i syslogd Simple Rotation # /sbin/init. d/syslogd stop # /sbin/init. d/syslogd start It will rename syslog. log to OLDsyslog. log in /var/adm/syslog directory. 03 November 2020 Webinar - Dusan Baljevic 22
HP-UX 11 i Other RC Cleanups • /etc/rc. config. d/clean_tmp for /tmp cleanup at boot CLEAR_TMP=1 • /etc/rc. config. d/clean CLEAN_ADM=1 CLEAN_UUCP=1 /var/adm/sulog /var/adm/diaglog /var/adm/messages renamed to OLD* • /etc/rc. config. d/clean_uucp uuclean(1 m) at boot CLEAN_UUCP=1 03 November 2020 Webinar - Dusan Baljevic 23
HP-UX 11 i Examples of Log Directories and Files • Examples of log files that can grow out of bounds: /var/spool/lp /var/adm/lp /var/opt/perf/datafiles lost+found directories in top-level of each file system /var/adm/diag /var/opt/ignite /var/stm/logs/sys * /var/adm/wtmps /var/adm/btmp /var/adm/sw (others truncated for the sake of brevity) 03 November 2020 Webinar - Dusan Baljevic 24
HP-UX 11 i auto_parms. log • /etc/auto_parms. log is updated by auto_parms(1 m) command that handles first-boot configuration (setting of unique system “initial identity parameters”), and ongoing management of DHCP leases • auto_parms(1 m) saves old copy into /etc/auto_parms. log. old 03 November 2020 Webinar - Dusan Baljevic 25
HP-UX 11 i rc. log • Run Command (RC) scripts update /etc/rc. log at boot time • At reboot, previous version of /etc/rc. log is renamed to /etc/rc. log. old 03 November 2020 Webinar - Dusan Baljevic 26
HP-UX 11 i EMS Logs • The EMS log files in /etc/opt/resmon/log are limited to 500 KB in size and are then moved to <logfile>. old. The previous *. old gets lost • The limit of 500 KB per logfile can be removed by creating the file /etc/opt/resmon/unlimited_log • Be careful with creating the unlimited_log. Growing EMS log files can easily fill up root file system 03 November 2020 Webinar - Dusan Baljevic 27
HP-UX 11 i Glance and Measure. Ware Logs • /var/opt/perf/parm is read by both the Glance. Plus product and the Measure. Ware products. Glance uses only the Application definitions size global=10, application=10, process=20, device=10, transaction=10 The sizes are in MB • The logfiles are stored in /var/opt/perf/datafiles directory 03 November 2020 Webinar - Dusan Baljevic 28
HP-UX 11 i Integrity VM Driver Log File • /var/opt/hpvm/common/hpvm_mon_log is limited to 1024 KB by default. When the log file grows larger than this, it is copied to a new file (hpvm_mon_log. $time) and an empty one is created for the new log • To allow this log file to grow larger than 1024 KB, include the following line in /etc/rc. config. d/hpvmconf VMMLOGSIZE=10420 # In KB Then, restart the daemon: # kill –HUP `cat /var/run/hpvmmonlogd. pid` 03 November 2020 Webinar - Dusan Baljevic 29
HP-UX 11 i Integrity VM Guest Log File • /var/opt/hpvm/guests/guest_name/log file records guest start and stop information. These log files can grown very large To close the current log file, rename it, and open a new one: # hpvmconsole rec -rotate 03 November 2020 Webinar - Dusan Baljevic 30
HP-UX 11 i SMH * # cat /opt/hpsmh/conf. common/smhpd. xml <? xml version="1. 0" encoding="UTF-8"? > <system-management-homepage> <admin-group></admin-group> <operator-group></operator-group> <user-group></user-group> <allow-default-os-admin>True</allow-default-os-admin> <anonymous-access>False</anonymous-access> <localaccess-enabled>False</localaccess-enabled> <localaccess-type>Anonymous</localaccess-type> <trustmode>Trust. By. Cert</trustmode> <xenamelist></xenamelist> <ip-restricted-logins>False</ip-restricted-logins> <ip-restricted-include></ip-restricted-include> <ip-restricted-exclude></ip-restricted-exclude> <ip-binding>False</ip-binding> <ip-binding-list></ip-binding-list> <rotate-logs-size>N</rotate-logs-size> </system-management-homepage> 03 November 2020 Webinar - Dusan Baljevic 31
HP-UX 11 i TCB Auditing • It records instances of access by subjects to objects and allows detection of any (repeated) attempts to bypass the protection mechanism and any misuses of privileges • audsys allows the user to start or halt the auditing system, to specify the auditing system "current" and "next" audit files (and their switch sizes), or to display auditing system status information. The "current" audit file is the file to which the auditing system writes audit records. When the "current" file grows to either its Audit File Switch (AFS) size or its File Space Switch (FSS) size (see audomon), the auditing system switches to write to the "next“ audit file # audsys Auditing system is currently on current file: /var/adm/audit/audfile 1 next file: /var/adm/audit/audfile 2 statistics- afs Kb used Kb avail % fs Kb used Kb avail % current file: 10000 0 100 4825088 963704 80 next file: 10000 0 100 4825088 963704 80 03 November 2020 Webinar - Dusan Baljevic 32
HP-UX 11 i HIDS log files increase rapidly. However, the Configuration Change Console agent keeps log files truncated to save disk space. To ensure that the log files do not increase in file size while the agent is not running, run a script to periodically truncate the HIDS log files. A sample script to manage HIDS log files is provided. This script should be run from the crontab: #!/bin/sh filesize=`/bin/ls -l /var/opt/ids/alert. log | /bin/awk '{print $5}'` if [ "$filesize" -gt "5000000" ] then mv /var/opt/ids/alert. log_Dec_2008 fi rm /var/opt/ids_1* Sample entry to configure the crontab to run every hour where the bold letters are replaced by the actual path of the trunclog. sh file: 0 * * /<location of script>/trunclog. sh 2>/dev/null 2>&1 03 November 2020 Webinar - Dusan Baljevic 33
HP-UX 11 i Service. Guard Package Log File SCRIPT_LOG_FILE (SG 11. 17+) A new package attribute that allows a name to be assigned to a package log file Necessary for support of multiple packages sharing a common package control script Legacy Package Configuration SCRIPT_LOG_FILE /etc/cmcluster/pkga. log Modular Package Configuration script_log_file $SGRUN/log/$SG_PACKAGE. log 03 November 2020 Webinar - Dusan Baljevic 34
HP-UX 11 i ulimit # ulimit -a time(seconds) unlimited file(blocks) unlimited data(kbytes) 1048576 stack(kbytes) 8192 memory(kbytes) unlimited coredump(blocks) 4194303 nofiles(descriptors) 2048 03 November 2020 Webinar - Dusan Baljevic 35
HP-UX 11 i v 3 coreadm * # coreadm global core file pattern: init(1 M) core file pattern: global core dumps: disabled per-process core dumps: enabled global setid core dumps: disabled per-process setid core dumps: disabled 03 November 2020 Webinar - Dusan Baljevic 36
HP-UX cleanup - HP-UX patch cleanup utility # cleanup –c 1 The cleanup command provides functions useful when dealing with HP-UX patches. The cleanup command logs all information to /var/adm/cleanup. log. 03 November 2020 Webinar - Dusan Baljevic 37
HP-UX savecrash utility /etc/rc. config. d/savecrash CHUNK_SIZE Size of single crash image file (how big you want each of image. n. x, image. n. x+1, etc. to be). If not specified, savecrash will choose one based on the physical memory size of the system. Can be specified in bytes (b), kilobytes (k), megabytes (m), or gigabytes (g). The default unit is KB. See savecrash(1 M) “–s” option for size constraints. COMPRESS: Whether you want the kernel and crash image files to be compressed. 03 November 2020 Webinar - Dusan Baljevic 38
HP-UX Alternative Log File Tools (part 1/3) • Old but maybe still applicable bundle (needs to be tested): http: //hpux. cs. utah. edu/hppd/hpux/Sysadmin/logrotate-2. 5/ • Shell script logrotate: http: //iain. cx/src/logrotate/ • Scripts based on Perl modules like Logfile-Rotate • Perl-Logrotate: http: //freshmeat. net/projects/perl-logrotate 03 November 2020 Webinar - Dusan Baljevic 39
HP-UX Alternative Log File Tools (part 2/3) • newsyslog project (old and possibly obsolete): http: //www. weird. com/~woods/projects/newsyslog. html • logtrim by Bill Hassell (released in HP ITRC forum several years ago): http: //forums 11. itrc. hp. com/service/forums/questionanswer. d o? thread. Id=1053445&admit=109447626+1221799837763+2 8353475 • Replace standard syslog daemon with Syslog-NG and SQL database: http: //www. balabit. com/network-security/syslogng/features/detailed 03 November 2020 Webinar - Dusan Baljevic 40
HP-UX Alternative Log File Tools (part 3/3) • smartlog (very old bundles for HP-UX 10. 20 and 11. 00 only): http: //gatekeep. cs. utah. edu/hppd/hpux/Sysadmin/smartlog 3. 5/ • Many other Shell scripts, for example: http: //www. zazzybob. com/bin/logrevolver. sh. html • Log. Watch: http: //www 2. logwatch. org: 81/ 03 November 2020 Webinar - Dusan Baljevic 41
HP-UX Syslog-NG • Syslog-NG and SQL database (My. SQL, Microsoft SQL (MSSQL), Oracle, Postgre. SQL, SQLite) • Log rotation based on output filenames - Log output filenames can be based on templates names which support macro expansion. For example, if the output filename template contains the month macro, a new filename will created each month • Often, syslog-ng is used for log file consolidation (centralized management) 03 November 2020 Webinar - Dusan Baljevic 42
Conclusion • Log file management is mostly managed reactively • Majority of Unix admins I meet are not aware of OS-native tools that are designed for log file administration 03 November 2020 Webinar - Dusan Baljevic 43
Thank You! Dusan Baljevic Sydney, Australia © 2008 Dusan Baljevic The information contained herein is subject to change without notice
- Jika log 2 = 0 301 dan log 3 = 0 477
- Jika log 100 = x tentukan nilai x
- Bentuk sederhana dari 5log 10 + 5log 50 - 5log 4 adalah
- Log 50 - log 5 + 2 log 2
- Diketahui log 2 = 0 301 dan log 5 = 0 699
- Nilai dari ³ log 27
- Jika log 3=0 477 dan log 5=0 699 maka log 45 adalah
- Penyelesaian pertidaksamaan log(x-4)+log(x+8) log(2x+16)
- File-file yang dibuat oleh user pada jenis file di linux
- Unix file system
- Unix file name
- Unix
- Fifo file api in unix
- Amelia belinda
- Unix file api
- A fast file system for unix
- Unity connection wav file format
- Physical image vs logical image
- Power law log log plot
- Power law log log plot
- Log a x log a
- Loga mn
- Experiment 343
- Ln
- Persamaan 7 log 217 + 7 log 31 ialah
- Log(a+b)
- How to type log in desmos
- Modelo log lin
- Exponential functions formula
- Log m-log n
- Log multiply log
- Condense the logarithmic expression
- Hyper log
- Log files stata
- Kyle hailey
- Fungsi sistem file
- Distributed file system definition
- An html file is a text file containing small markup tags
- In a file-oriented information system, a transaction file
- Unix process management
- Buddy algorithm in linux
- Unity datatable
- Module 4 operating systems and file management
- Is unix and linux same
- What is unix and linux