IBM Software Group IBM Tivoli Workload Scheduler Monitor

IBM Software Group | Tivoli software Abstract This session discusses function of the mon_msg. sh script, requirements, operation and implementation to facilitate monitoring of UNIX Tivoli Workload Scheduler. msg files and TWS file system disk space availability. It will focus on: ۰ Script function ۰ Script requirements ۰ Customizable Variables ۰ Non-customizable Variables ۰ Operation ۰ Implementation ۰ Troubleshooting Note: A demonstration of the mon_msg. sh script will be also presented at the end of this session. 2 TWS STE | Monitor TWS. msg files and Filesystem © 2008 IBM Corporation

IBM Software Group | Tivoli software mon_msg. sh Function The mon_msg. sh script is used to monitor the size of TWSHome/*. msg, TWSHome/pobox/*. msg files and TWS filesystem available disk space on a UNIX workstation. Script Execution: <path>/mon_msg. sh <tws_user> Script uses user a defined ratio that monitor records in. msg files. When current record ratio exceeds defined ratio, script can be configured to: § Auto increase size of. msg file using a user defined or default percentage. § Send notification that. msg file ratio was exceeded § Perform a user defined task. Script will monitor available disk space for TWS filesystem. If the current available number of kbytes drops below the defined or default value, script can send notification or perform another task. 3 TWS STE | Monitor TWS. msg files and Filesystem © 2008 IBM Corporation

IBM Software Group | Tivoli software Script Requirements § TWS user specified at install must be specified when mon_msg. sh script is executed. User must exist in the TWSRegistry. dat file. § The /tmp directory must exist on workstation, since /tmp directory is where temporary files will be generated. § Script may exist in any directory. § Script must have execute permissions for TWS user for the TWS instance. § If script will be executed for multiple TWS instances that exist on workstation then access permissions will need to be adjusted accordingly. § If script is defined as a TWS job then script path must be specified in the job definition. § Entries in the TWSRegistry. dat and “thiscpu” variable in TWSHome/localopts file must be valid. 4 TWS STE | Monitor TWS. msg files and Filesystem © 2008 IBM Corporation

IBM Software Group Customizable Variables © 2008 IBM Corporation

IBM Software Group | Tivoli software mon_msg Disk Space Monitor Variables Variable: FILE_SYS_AVAIL_SPACE_LIMIT Minimum low limit setting. This number will be used as base to determine if available disk space of the TWS filesystem has dropped below the specified value. Use whole numbers, the value should be in k-bytes. The value specified should allow TWS/UNIX administrator sufficient time to make adjustments before TWS runs out of disk space, causing TWS processes to terminate and possibly compromising the Symphony file. FILE_SYS_AVAIL_SPACE_LIMIT=<kbytes>. Default value (1500000): FILE_SYS_AVAIL_SPACE_LIMIT=15000000 Note: 15000000 KB is about 14648. 4 MB or 14. 3 GB Variable: FILE_SYS_INVOKE_NOTIFY Value to determine if notification is required when available disk space for TWS filesystem has dropped below specified limit. Set value to "yes" to invoke notification, any other value will default to "no". FILE_SYS_INVOKE_NOTIFY=<yes or no>. Default value: FILE_SYS_INVOKE_NOTIFY=yes 6 TWS STE | Monitor TWS. msg files and Filesystem © 2008 IBM Corporation

IBM Software Group | Tivoli software mon_msg Disk Space Monitior Variables Variable: FILE_SYS_NOTIFY_CONTACT Value to identify contact to receive notification when TWS filesystem drops below the free disk space value specified in variable FILE_SYS_AVAIL_SPACE_LIMIT. Default action is to send email to root user if variable FILE_SYS_INVOKE_NOTIFY is set to yes. An email list could be specified if several users should be notified by separating users with a comma ", ". FILE_SYS_NOTIFY_CONTACT=<email_id> or <cell_phone_number@carrier> (see Cell Phone SMS Text Messaging). Default value: FILE_SYS_NOTIFY_CONTACT=root Example: FILE_SYS_NOTIFY_CONTACT=root, 210 -555 -1234@txt. att. net, 512 -5552345@messaging. sprintpcs. com 7 TWS STE | Monitor TWS. msg files and Filesystem © 2008 IBM Corporation

IBM Software Group | Tivoli software Notification via Cell Phone SMS Text Messaging Major US cellular carriers use the 10_digit_number@cellcarrier_domain. com format for SMS to text capable cell phones, with a limit of 160 characters in the subject and message body (total). You will need to determine limitations for subject and body of email for the respective carrier. Carrier------> Alltel-------> ATT-----> Nextel ------> Sprint-------> Sun. Com-------> T-mobile-----> Verizon------> Virgin Mobile> Voice. Stream--> Send Email to phonenumber@. . @message. alltel. com @txt. att. net @messaging. nextel. com @messaging. sprintpcs. com @tms. suncom. com @tmomail. net @vtext. com @vmobl. com @voicestream. net Note: Cell phone must be able to receive text messages and carrier may have limitations. 8 TWS STE | Monitor TWS. msg files and Filesystem © 2008 IBM Corporation

IBM Software Group | Tivoli software mon_msg. MSG Files Variable: THRESH_PER Upper limit percentage setting. This number will be used as ratio to determine when a. msg file has exceeded its upper limit. Use whole numbers, do not use a decimal point. The calculation will convert to decimal value. THRESH_PER=<two digit value>. Default value (85%): THRESH_PER=85 Variable: AUTO_INCREASE_MSG Value to determine if. msg file should be increased automatically when value of a. msg file is at or greater than upper limit percentage specified in variable THRESH_PER. The default is AUTO_INCREASE_MSG=no. Set value to "yes" to invoke auto adjust. Any other value will default to "no". Increasing of. msg file will permit. msg file to increase in size as more records are added which means that it will require additional disk space. TWS filesystem must be able to handle increased size. AUTO_INCREASE_MSG=<yes or no> Default value: AUTO_INCREASE_MSG=no 9 TWS STE | Monitor TWS. msg files and Filesystem © 2008 IBM Corporation

IBM Software Group | Tivoli software mon_msg. MSG Files Variable: AUTOMSG_INCR_PER Percentage of increase for. msg files that exceed the upper limit percentage specified in variable THRESH_PER. AUTOMSG_INCR_PER. The increase of. msg file is performed by TWS evtsize binary. The use of this binary permits the increase while TWS is still active. Use whole numbers, do not use a decimal point. The calculation will convert to decimal value. AUTOMSG_INCR_PER = < 1 or 2 digit value> Default value (20%): AUTOMSG_INCR_PER=20 Variable: MSG_INVOKE_NOTIFY Value to determine if notification is required when of upper limit of. msg file has been exceeded. Set value to "yes" to invoke notification, any other value will default to "no". MSG_INVOKE_NOTIFY=<yes or no>. Default value: MSG_INVOKE_NOTIFY=yes 10 TWS STE | Monitor TWS. msg files and Filesystem © 2008 IBM Corporation

IBM Software Group | Tivoli software mon_msg. MSG Files Variable: MSG_NOTIFY_CONTACT Value to identify contact to receive notification when upper limit of. msg file has been exceeded. Default action is to send email to root user if variable MSG_INVOKE_NOTIFY is set to yes. An email list could be specified if several users should be notified by separating users with a comma ", ". MSG_NOTIFY_CONTACT=<email_id> or <cell_phone_number@carrier> (see Cell Phone SMS Text Messaging). Default value: MSG_NOTIFY_CONTACT=root Example: MSG_NOTIFY_CONTACT=root, 210 -555 -1234@txt. att. net, 512 -5552345@messaging. sprintpcs. com 11 TWS STE | Monitor TWS. msg files and Filesystem © 2008 IBM Corporation

IBM Software Group | Tivoli software Notification via Cell Phone SMS Text Messaging Major US cellular carriers use the 10_digit_number@cellcarrier_domain. com format for SMS to text capable cell phones, with a limit of 160 characters in the subject and message body (total). You will need to determine limitations for subject and body of email for the respective carrier. Carrier------> Alltel-------> ATT-----> Nextel ------> Sprint-------> Sun. Com-------> T-mobile-----> Verizon------> Virgin Mobile> Voice. Stream--> Send Email to phonenumber@. . @message. alltel. com @txt. att. net @messaging. nextel. com @messaging. sprintpcs. com @tms. suncom. com @tmomail. net @vtext. com @vmobl. com @voicestream. net Note: Cell phone must be able to receive text messages and carrier may have limitations. 12 TWS STE | Monitor TWS. msg files and Filesystem © 2008 IBM Corporation

IBM Software Group Non-customizable Variables © 2008 IBM Corporation

IBM Software Group | Tivoli software Non-Customizable options Changes to the following variables will cause problems with the script: NEW_AUTOMSG_INCR_PER=1. ${AUTOMSG_INCR_PER} TWS_REGISTRY_PATH="/etc/TWS"; export TWS_REGISTRY_PATH TWS_REGISTRY_FILE="${TWS_REGISTRY_PATH}/TWSRegistry. dat"; export TWS_REGISTRY_FILE AUTO_INCREASE_MSG=ècho ${AUTO_INCREASE_MSG}|tr "[: upper: ]" "[: lower: ]"`; export AUTO_INCREASE_MSG MSG_INVOKE_NOTIFY=ècho ${MSG_INVOKE_NOTIFY}|tr "[: upper: ]" "[: lower: ]"`; export MSG_INVOKE_NOTIFY FILE_SYS_INVOKE_NOTIFY=ècho ${FILE_SYS_INVOKE_NOTIFY}|tr "[: upper: ]" "[: lower: ]"`; export FILE_SYS_INVOKE_NOTIFY twsuser=$1; export twsuser MSG_INCREASE_MESSAGE=" "; export MSG_INCREASE_MESSAGE 14 TWS STE | Monitor TWS. msg files and Filesystem © 2008 IBM Corporation

IBM Software Group Notification Modules (tasks) © 2008 IBM Corporation

IBM Software Group | Tivoli software Notification Module The notification modules may be customized to perform any task or tasks. The current defined task for. msg and available disk space monitoring is to send an email to a user defined user. The notification modules may be customized by user to generate trouble tickets or to generate events that may be monitored by other applications. The section in red text in the following two slides are where task or action may be customized. NOTE: User customizations will not be supported by TWS L 2 support. 16 TWS STE | Monitor TWS. msg files and Filesystem © 2008 IBM Corporation

IBM Software Group | Tivoli software MSG Notification Module test -f /tmp/msg_max. $$ if [ "$? " = "0" ] then if [ "${MSG_INVOKE_NOTIFY}" = "yes" ] then #This section may be customized to perform another task #other than sending out an email #The following message will be sent. If script is being executed #from within a TWS job the ${UNISON_STDLIST} ${UNISON_JOB} #information will be included. #The message generated will depend it script was executed via TWS. if [ "${tws_job}" = "y" ] then mail ${MSG_NOTIFY_CONTACT} <<-! WARNING: else WARNING: One or more. msg files have exceeded their upper limit of ${THRESH_PER}% on TWS workstation ${TWSCPUNAME}. Please review stdlist ${UNISON_STDLIST} for job ${UNISON_JOB}. Files that exceeded threshold: ${MSG_MAX} ${MSG_INCREASE_MESSAGE} ! mail ${MSG_NOTIFY_CONTACT} <<-! WARNING: One or more. msg files have exceeded their upper limit of ${THRESH_PER}% on TWS workstation ${TWSCPUNAME}. fi 17 fi fi Files that exceeded threshold: ${MSG_MAX} ${MSG_INCREASE_MESSAGE} ! TWS STE | Monitor TWS. msg files and Filesystem © 2008 IBM Corporation

IBM Software Group | Tivoli software Sample Warnings – Script stdout and email message Disk Space threshold limit exceeded WARNING: Available disk space of file system /tws for TWS Home directory /tws/m 821 a is 609173 kbytes which is less than the defined threshold limit of 1500000 kbytes. Please increase size of file system /tws for TWS Home directory /tws/m 821 a before TWS file system runs out of disk space otherwise TWS operations may be compromised. TWS file system info: /dev/vg 00/lvol 9 16384000 15760057 609173 96% /tws. msg file record ratio exceeded, . msg not auto-increased WARNING: One or more. msg files have exceeded their upper limit of 85% on TWS workstation m 821 a. Files that exceeded threshold: Current warning level is set at 85% of 30000 records. Message file /tws/m 821 a/pobox/M 82 A. msg on m 821 a is at 28896 records i. e. . 96%. File limit size should be increased with /tws/m 821 a/bin/evtsize command before TWS operations are compromised. TWS file system info: /dev/vg 00/lvol 9 16384000 15760057 609173 96% /tws. msg file record ratio exceeded, . msg was auto-increased WARNING: One or more. msg files have exceeded their upper limit of 85% on TWS workstation m 821 a. Files that exceeded threshold: Current warning level is set at 85% of 30000 records. Message file /tws/m 821 a/pobox/M 82 A. msg on m 821 a is at 28896 records i. e. . 96%. AWSDEK 702 I File /tws/m 821 a/pobox/M 82 A. msg maximum size changed to 36000 bytes /tws/m 821 a/pobox/M 82 A. msg was increased by 20%. TWS file system info: /dev/vg 00/lvol 9 16384000 15760050 609180 96% /tws 18 TWS STE | Monitor TWS. msg files and Filesystem © 2008 IBM Corporation

IBM Software Group | Tivoli software Script Operation The script will verify that the specified user exists before proceeding with the execution of the script The values specified in mon_msg. sh script will determine operation and tasks performed by the script. The script will terminate immediately if specified user is not valid. The script will generate temporary files in the /tmp directory. Generated files will have a prefix of msg_ and include the process id/pid as an extension. These generated files will be removed as soon as the script completes. The increase of the. msg file is performed by the TWS evtsize binary. The use of this binary permits the increase of. msg file while TWS is still active. The script will display information for. msg files that exceed the defined upper limit ratio of. msg file. Displayed information will include information about filesystem where TWS instance is located. Display will be written to stdout. If script is executed from within a TWS job, output will be displayed in the job stdlist. 19 TWS STE | Monitor TWS. msg files and Filesystem © 2008 IBM Corporation

IBM Software Group | Tivoli software mon_msg Implementation Procedures Script may be executed from UNIX prompt or as a TWS defined job. User will need to determine how often script will launch and if job is defined with “every” dependency. To Execute From UNIX Prompt: 1. Edit mon_msg. sh and customize variables. 2. Change permission for script to include execute permissions. 3. Execute the mon_msg. sh script mon_msg. sh <tws_user_name> Where: tws_user_name is TWS user for workstation. To Execute as a TWS Job. 1. Edit mon_msg. sh and customize variables. 2. Change permission for script to include execute permissions. 3. Define a TWS job that executes the mon_msg. sh script. The logon user should be same user specified for script execution. 20 TWS STE | Monitor TWS. msg files and Filesystem © 2008 IBM Corporation

IBM Software Group | Tivoli software mon_msg Implementation Procedures 4. The path must be provided for the mon_msg. sh script. 5. Create a job stream and include the job with an “every” dependency. The “every” dependency will determine how often the job will execute. The job stream should be scheduled to run on a daily basis (on everyday). Job definition: M 821 A#MON_MSG SCRIPTNAME "/tws/m 821 a/scripts/mon_msg. sh m 821 a" STREAMLOGON "m 821 a" DESCRIPTION ". msg monitor script" RECOVERY STOP Job stream definition: SCHEDULE M 821 A#MON_MSG_SCH ON EVERYDAY : M 821 A#MON_MSG EVERY 60 END 21 TWS STE | Monitor TWS. msg files and Filesystem © 2008 IBM Corporation

IBM Software Group | Tivoli software Troubleshooting The script has specific requirements for execution. Review/verify the following: 1. Script mon_msg. sh must be executable. 2. Valid TWSuser must be specified when script is executed. Example: <path>/mon_msg. sh <twsuser> 3. The /tmp directory must exist. 4. The /etc/TWSRegistry. dat file must exist and include the TWSuser. 5. The “thiscpu” variable in the TWSHome/localopts file must have correct TWS workstation name. 6. Verify that TWS filesystem has enough disk space. 7. UNIX mail must be able to send out emails. 22 TWS STE | Monitor TWS. msg files and Filesystem © 2008 IBM Corporation

IBM Software Group Any Questions ? ? TWS 8. 4. 0 © 2008 IBM

IBM Software Group | Tivoli software Available Disk Space Monitoring Notification Module test -f /tmp/msg_file_sys_exceeded. $$ if [ "$? " = "0" ] then if [ "${FILE_SYS_INVOKE_NOTIFY}" = "yes" ] then #This section may be customized to perform another task #other than sending out an email #The following message will be sent. If script is being executed #from within a TWS job the ${UNISON_STDLIST} ${UNISON_JOB} #information will be included. #The message generated will depend it script was executed via TWS. if [ "${tws_job}" = "y" ] then mail ${FILE_SYS_NOTIFY_CONTACT} <<-! WARNING: ${TWSCPUNAME} filesystem has exceeded their upper limit. Available disk space of file system ${mounted_file_system} for TWS Home directory ${TWSPATH} is ${available_disk} kbytes which is less that the defined which is less than the defined threshold limit of ${FILE_SYS_AVAIL_SPACE_LIMIT} kbytes. else Please increase size of file system ${mounted_file_system} for TWS Home directory ${TWSPATH} before TWS file system runs out of disk space otherwise TWS operations may be compromised. TWS file system info ${TWS_file_sys} ! mail ${FILE_SYS_NOTIFY_CONTACT} <<-! WARNING: ${TWSCPUNAME} filesystem for TWS directory has exceeded specified upper limit. WARNING: ${TWSCPUNAME} file system ${mounted_file_system} for TWS Home directory ${TWSPATH} is ${available_disk} kbytes which is less than the defined threshold limit of ${FILE_SYS_AVAIL_SPACE_LIMIT} kbytes. Please increase size of file system ${mounted_file_system} for TWS Home directory ${TWSPATH} before TWS file system runs out of disk space otherwise TWS operations may be compromised. TWS file system info ${TWS_file_sys} fi 24 fi fi ! TWS STE | Monitor TWS. msg files and Filesystem © 2008 IBM Corporation