Korn Shell Script Writing HPC Science Technology Support

Korn Shell Script Writing & HPC Science & Technology Support Group High Performance Computing OSC (Ohio Supercomputer Center) 1224 Kinnear Road Columbus, OH 43212 1

Table of Contents • Introduction – – – • What is a shell script? Script Programming Language Why use shell scripts? Korn shell scripts References Sampling of Script Features A simple script A more versatile Script Creating Files within a Script A documented Script Basics – Creating a shell script – Executing a shell script – Echoing script commands Script Variables – User-defined Variables – Storing a Value into a Variable – – – The Evolution of a Script – – – • • Assignment Operator = • Read Command • Command Substitution Null Variables Special Shell Variables Positional Parameters Referencing Variables Parameter Substitution Exercises Decision-Making statements – – – – Flow Control Exit Status If-Then-Else command Conditional Tests Logical Operators Illustrative Examples Case statement Exercises 2

Table of Contents • Looping Statements – – – • Arithmetic Operations Operator Precedence Numerical Bases String vs Integer Variables Exercises Script I/O – – – Shell Redirection Operators "Here-is" Files print read Quoting Exercises Communicating with Running Scripts – Sending Signals – How Scripts Receive Signals • Script Functions • HPC Designed Scripts Working with Integers – – – • for Statement select Statement while Statement Until Statement Exercises • – – – Why Use Functions? Syntax & Use Auto-Loading Function Definition Location Command Precedence – – Timing Loops Debugging Options Batch Jobs Performance Tool Preprocessing – Multiple Tools runs – Multiple Tool Reports – File Renaming 3

Introduction • • • What is a shell script? Script Programming Language Why use shell scripts? Korn Shell scripts References 4

What is a shell script? • A shell script is a collection of Unix commands entered into a file. When the file (script) is executed, the commands are executed in order. • Running a shell script is EXACTLY equivalent to interactively entering each command • For example, say a user always does the same four steps before printing out a file: making the file owner- writable, using vi to check/change the file, printing it out, and then making the file unwritable (for safety reasons) • An interactive session of a user following this procedure for the file comp. txt is shown on the next page 5

Interactive editing and printing 1. $ chmod u+w comp. txt 2. $ vi comp. txt 3. Here is a table summarizing OSC computers as of today. 4. Supercomputer # Processors Proc Speed 5. ---------------6. Itanium Cluster 124 900 MHz 7. Cray SV-1 16 300 MHz 8. Origin 2000 64 300 MHz 9. IA-32 Intel 128 550 MHz 10. Sun Fire 6800 24 900 MHz 11. "comp. txt" 11 lines, 535 characters 12. $ lpr -Pps 1 comp. txt Ideal Calc. ------223. 2 GFLOPS 19. 2 GFLOPS 38. 4 GFLOPS 70. 4 GFLOPS 43. 2 GLOPS 13. $ chmod u-w comp. txt 6

Making the viprint script • To make the previous editing and printing procedure into a script, all the user has to do is type the same commands they use interactively into a file • Say the file is called viprint, it looks like this: $ cat viprint chmod u+w comp. txt vi comp. txt lpr -Pps 1 comp. txt chmod u-w comp. txt • After viprint is made executable, whenever the user wants to execute these four commands they simply type viprint as if it was a “normal” Unix command $ viprint 7

Script Programming Language • The real power is writing shell script comes from using special shell commands and variables that essentially make your script act like a program • In this course, you will learn about the programming statements which can be used in scripts, including: – Initializing and using your own script variables. Both single valued and arrays – Performing operations with variables – Decision-making statements (if-then-else, etc) – Looping statements (for, etc) – Defining and using functions in scripts – Script I/O commands and procedures • Above bullet reads like a “table of contents” for a text on a “traditional” programming languages such as Fortran or C, although … • Script programming has less capabilities and commands 8

Why use shell scripts? • • • Convenience: Run a number of Unix commands by typing a single script name Command log: Script can permanently contain a detailed set of commands with numerous options. (If interactive, complex command is gone) Usefulness: Write a script with programming commands to give the script a number of capabilities/options Take “full” use of what the shell has built-in to offer you Considering a script as a program, no compiler, loader, or libraries needed to make it executable Portability: If a script works in the shell on your machine it will work on the same shell on a different machine Easy to debug: Small number of programming statements which are relatively simple Easy to modify, expand, and customize: Just type in the new commands Run-type interaction: Can send signals to running scripts 9

Korn shell scripts • Several types of Unix shells exist, each with their own script programming languages: – Bourne (sh), Born Again shell (bash), Korn shell – C shell (csh), TENEX C shell (tcsh) • Features you would expect from a well-written shell: – Convenient interactive features: command history, command-line editing, and filename completion – Powerful scripting capabilities including reliability and efficiency • This course will discuss Korn shell scripting – It has all the convenient interactive features – It is backward compatible with the Bourne shell (the first shell) • Most scripts are sh • Start-up sequence for ksh same as sh only enhanced • Unix utilities and administration commands depend on sh – It does not have the scripting bugs that the C shell has 10

Korn shell scripts • What if I already use a different shell on my machine? – Could switch to Korn shell (use the chsh command if you can) – Keep your existing shell as your interactive shell and write your scripts as Korn shell scripts (Easily done: one line needed in the script) 11

References • Internet URLs – nacphy. physics. orst. edu/rubin/melanie/node 144. html (Tutorial) – gonzo. tamu. edu/csh. whynot. html (Bugs in C shell scripting) • Usenet Newsgroups – comp. unix. shell – comp. unix. questions • Anonymous ftp – athos. rutgers. edu/~patra/vi. html (shell-101. Beta. A. Z) – rtfm. mit. edu: /pub/usenet/news. answers/unix-faq/faq (Unix FAQ) • OSC Unix-Series workshops (oscinfo. osc. edu) – Basic Unix – Intermediate Unix – Introduction to Perl 12

The Evolution of a Script • • • Sampling of Scripting Features A Simple Script A More Versatile Script Creating Files within a Script A Commented Script 13

Sampling of Scripting Features • Next few slides will act as a course overview as we take a simple script and keep extending its capabilities • The more advanced scripts will demonstrate some of the capabilities of Korn Shell scripting • Don’t worry about understanding all the details at this point. A complete explanation of the scripting commands used is given in later chapters 14

A Simple Script • • In the script called triangle, a C program is compiled and then executed. The program calculates the area of the triangle and reads in the lengths of the sides from a file and outputs the result to another file. Here is an example of a user running the script: $ cat triangle cc -n 32 -cckr -O 3 -LNO -Ofast -TARG: madd=on -o area. c area < sides. in > area. dat cat area. dat rm area $ ls -l triangle -rwxr--r-1 dje appl 108 May 3 12: 09 triangle $ cat sides. in 8 3 7 $ triangle Area of Triangle is 10. 392305 15

A Simple Script • Limitation of the simple triangle script – Script only works for the one area program. What if you want to run other triangle-related programs with this same script ? 16

A More Versatile Script • • Scripts can take arguments on the command line when they are run. The first argument is passed to a variable with a special name: $1. We can use this feature to make the triangle script work for any program. Here is the resulting new script in action: $ cat triangle cc -n 32 -cckr -O 3 -LNO -Ofast -TARG: madd=on -o $1 $1. c $1 < sides. in > $1. dat cat $1. dat rm $1 $ triangle area Area of Triangle is 10. 392305 $ triangle perim Perimeter of Triangle is 18. 000000 17

A More Versatile Script • Limitation of versatile script: – Scripts should be as independent of possible of “outside” existing files (for portability and ease of use). Since the sides. in file is so short, we should create the file within the script 18

Creating Files Within a Script • The easiest method for creating a small “internal” file is to use “Here-is” Unix redirection. After it is used, this temporary file is removed: $ cat triangle if [[ $2 = debug ]]; then cc -g -o $1 $1. c print "$1 meant to be debugged" else cc -n 32 -cckr -O 3 -LNO -Ofast -TARG: madd=on -o $1 $1. c cat << EOF > sides. in 8 3 7 EOF $1 < sides. in > $1. dat cat $1. dat rm $1 sides. in fi $ triangle perim Perimeter of Triangle is 18. 000000 19

A Commented Script • A bad script writing style has been shown in all incarnations of the triangle script: the lack of comments. Comments are critical to another user (or you) in understanding what the script does. • The special “comment” on the first line of script actually specifies the shell to be used for the script commands. This is required for users whose interactive and script shells differ. • On the next page is the timing script just discussed but now with proper commenting 20

A Commented Script $ cat triangle #!/bin/ksh # Compiling C code found in first parameter cc -n 32 -cckr -O 3 -LNO -Ofast -TARG: madd=on -o $1 $1. c # Using redirection to make input file cat << EOF > sides. in 8 3 7 EOF # Using while loop to run the executable 17 times let i=0 while (( i < 17 )); do $1 < sides. in >> $1. dat let i=i+1 done # Checking output cat $1. dat # Clean up files no longer needed rm $1 sides. in 21

Script Basics • Creating a Shell Script • Executing a Shell Script • Echoing Script Commands 22

Creating a Shell Script • A shell script is an executable file which is executed by the shell line-by-line. It can contain the following: – UNIX commands – shell programming statements – comments • Create using editor of choice • Can include a #! construct in first line of script to override login shell – #!/bin/ksh uses Korn shell to execute script – #!/bin/csh uses C shell to execute script – Recommend always put this in every Korn shell script. Makes it clear what the file is. "file" Unix command will even identify it as a Korn Shell script • Name should be short and descriptive 23

Executing a Shell Script There are 3 ways to execute a shell script: 1. "dot" method $. scriptname 2. "just the name" method $ scriptname 3. in the background $ scriptname & 24

Executing a Shell Script • Method 1 runs the command as if you typed them in on the command line – No ksh subprocess created • Note that methods 2 and 3 require: – execute permission for scriptname chmod +x scriptname – current directory (. ) must be in PATH or else must use. /scriptname 25

Executing a Shell Script • Demonstration of execution methods: $ ps PID TTY 31612 ttyq 5 31631 ttyq 5 $ ls -l showsh -rw-r--r-1 dje $ cat showsh print Executing the ps $. showsh Executing the shell PID TTY 31612 ttyq 5 31628 ttyq 5 TIME CMD 0: 00 sh 0: 00 ps appl 43 Nov 25 1997 showsh shell script showsh TIME CMD 0: 00 sh 0: 00 ps 26

Executing a Shell Script $ chmod u+x showsh $ showsh Executing the shell PID TTY 31578 ttyq 5 31612 ttyq 5 31635 ttyq 5 script showsh TIME CMD 0: 00 ps 0: 00 sh 27

Echoing Script commands • For understanding or debugging scripts it is often convenient to see the script commands shown on the monitor along with the output they produce. • Accomplished using the set -x command at the beginning of the file • To turn off the echoing, make the command set +x (counterintuitive? ) • Also useful in batch job files and - on occasion interactively • On the next page a demonstration of the power of set -x is shown 28

Echoing Script commands $ see /homea/dje/ksh_script boba. osc. edu: 0. 0 a_opt. txt cba_opt. txt b_opt. txt fant 4. txt c_opt. txt item. intmod. txt $ cat see #!/bin/ksh pwd echo $DISPLAY ls *. txt $ cat see 2 #!/bin/ksh set -x pwd echo $DISPLAY ls *. txt item. set. 3. txt item. set. 4. txt no_opt. txt pid. txt showsh. txt trap. txt 29

Echoing Script commands $ see 2 + pwd /homea/dje/ksh_script + echo boba. osc. edu: 0. 0 + ls a_opt. txt b_opt. txt cba_opt. txt fant 4. txt item. intmod. txt item. set. 3. txt item. set. 4. txt no_opt. txt pid. txt showsh. txt trap. txt a_opt. txt cba_opt. txt item. set. 3. txt pid. txt b_opt. txt fant 4. txt item. set. 4. txt showsh. txt c_opt. txt item. intmod. txt no_opt. txt trap. txt 30

Script Variables • User-defined Variables • Storing a Value into a Variable – The assignment operator = – The read command – Command Substitution • • • Null Variables Special Shell Variables Positional Parameters Referencing Variables Parameter Substitution Exercises 31

User-defined Variables • The Korn shell includes the following types of variables: – User-defined variables – Special shell variables • User-defined variables can be initialized, used and changed from the command line or from within a shell script. – Not "declared" like in a programming language. Just type the name of the variable – Korn shell scripts use string and integer variables • A variable name can consist of the following: – letters, digits, and the underscore character – first character of a variable name must be a letter or an underscore character • A variable can be made read-only: once it is assigned a value, the value cannot be changed – "Archaic" method: readonly variable_name – Alternate method: typeset -r variable_name 32

Storing a Value into a Variable • There are several methods for variables to obtain values. j The assignment operator = • Enter the name that you have chosen for the variable followed by an equal sign and then the value that you want to store in the variable. • To use the a variable's value, put the $ operator in front of it's name. (Read the $ as "contents of") • The typeset command can also be used for variable assignment. More up-to-date procedure, typeset has many options. • To assign a value to an integer variable, the assignment command must be proceeded by the word let • On the next page, a script is shown that demonstrates all these approaches 33

Storing a Value into a Variable 1. $ cat assign 2. street=Elm 3. print street <-- Most common error when using variables 4. print $street 5. my_name="John Smith" 6. echo $my_name 7. neph=Chris Sellgren 8. print $neph 9. typeset card=ace; print $card 10. let num=57; echo $num 11. let num=num+3; print $num 12. $ assign 13. street 14. Elm 15. John Smith 16. assign 2. ksh[9]: Sellgren: not found 17. ace 18. 57 19. 60 34

Storing a Value into a Variable k Second method: read in a value for a variable • So far have used print and echo (archaic) commands for printing out all the strings following them and then a carriage return • There is a complimentary command read which input a value into variables. The user type in as many strings as there are variables on the read line, and then hits carriage return • On the next page is a demo script indicating how the read command works. 35

Storing a Value into a Variable $ cat input print Please enter a ship name read ship print Please enter your full name read fname lname print Captain $lname, welcome aboard the $ship echo May I call you $fname? print Enter an integer; read num let num=2*num echo Twice your number is $num $ input Please enter a ship name Redoubt Please enter your full name Horatio Hornblower Captain Hornblower, welcome aboard the Redoubt May I call you Horatio? Enter an integer 27 Twice your number is 54 36

Storing a Value into a Variable l Use command substitution. The string(s) that a Unix command returns may be put into a script variable • The following syntax is as follows: var_name=$(UNIX command) • As with all the script commands, command substitution can be done interactively as well • On the next page is a demo script illustrating this technique 37

Storing a Value into a Variable 136: oscb$ cat command dir=$(pwd) print I am in the directory $dir weekday=$(date +%A) print It is $weekday files=$(ls -lt | wc -l) print There a $files in this directory me=$(whoami) print I am $me computer=$(hostname) print My computer is called $computer logged_on=$(who | cut -f 1 -d' ') print These people are also on $computer: $logged_on 137: oscb$ command I am in the directory /home/dje It is Tuesday There a 277 files in this directory I am dje My computer is called oscb These people are also on oscb: osu 2796 opr 006 paul ysu 039 osu 2796 jimg mmittal osu 2224 srb cls 038 osu 1938 jeff dje 38

Null Variables A variable can be set to a null value, even if previously assigned, using any of the following methods. There should be no spaces preceding or following the equal sign. If the contents of null character variables are printed out, a blank appears. If the contents of a “nullled” integer variable is outputted, a 0 is displayed. $ name='' $ name="” $ let num= $ unset varname 39

Null Variables All variables that don't exist are assumed null unless set -o nounset is used. Then the shell will indicate an error when an undefined variable is encountered $ unset name $ set -o nounset $ print $name ksh: name: parameter not set $ set +o nounset 40

Special Shell Variables • In addition to your own user-defined variables, other variables defined by the shell for the user are called special shell variables and are most useful • These variables are set and updated automatically by the shell. Their values cannot be changed, but they may be referenced. • The special shell variables of most use are identified by the following symbols: $# $$? $$ $! $0 $* 41

Special Shell Variables • The variable $# contains the number of arguments typed on the command line. $ cat numargs print The number of arguments is $# $ numargs The number of arguments is 0 $ numargs 1 2 3 4 5 The number of arguments is 5 $ numargs "Hello World" The number of arguments is 1 42

Special Shell Variables • The variable $- contains the shell flags (options) of the current shell. $ print $isum $ set +u $ echo $ism 43

Special Shell Variables • The variable $? contains the exit status of the last command. If the exit status is 0, the command ran successfully. If it is non-zero, there was a problem $ ls file 1 data account. txt $ rm file 1 $ print $? 0 $ rm dataa: No such file or directory $ print $? 2 • We will see exit status again. It will be used to define true and false. Also, the user can set the exit status in their script by using the exit command 44

Special Shell Variables • The variable $$ contains the Process ID (PID) of the current shell process. $ cat pid #!/bin/ksh ps print $$ $ pid PID 34738 34744 34765 $. pid PID 34743 34744 TTY ttyq 5 TIME 0: 00 CMD ps sh pid TTY ttyq 5 TIME CMD 0: 00 ps 0: 00 sh 45

Special Shell Variables • The variable $! contains the process ID number of the last command sent to the background. $ sleeper & [1] 29144 $ ps PID TTY TIME 29144 ttyq 5 0: 00 29150 ttyq 5 0: 00 29167 ttyq 5 0: 00 $ compress moby_dick & [2] 29136 $ print $! 29136 CMD sleeper sh ps sleep 46

Special Shell Variables • The variable $0 contains the name of the script currently being execute $cat old_script print The name of this script is $0 $old_script The name of this script is old_script $mv old_script new_script $new_script The name of this script is new_script • The variable $* contains all the arguments on the command line. $ cat args print The arguments are: $* $ args bob dave The arguments are: bob dave 47

Positional Parameters • For a script, the command line arguments are also called positional parameters. Each positional parameters refers to one individual argument on the command line. The ten positional parameters available to the script writer are referenced as follows: $1 $2 $3 $4 $5 $6 $7 $8 $9 • The parameter $1 contains the first argument, $2 the second argument, and so on. Consider the following script: $ cat print print parms Arg 1 Arg 2 Arg 3 Arg 4 Arg 5 is: is: is: $1 $2 $3 $4 $5 48

Positional Parameters $ parms 2001 A Space Odyssey Arg 1 is: 2001 Arg 2 is: A Arg 3 is: Space Arg 4 is: Odyssey Arg 5 is: $ parms "Space, the final frontier" Arg 1 is: Space, the final frontier Arg 2 is: Arg 3 is: Arg 4 is: Arg 5 is: 49

Positional Parameters Use the set command to change positional parameters. It replaces existing positional parameters with new values $ cat newpos print starting args are $* print number of args is $# print arg 1 is $1 print arg 2 is $2 set NCC 1701 Enterprise print new args are $* print number of args is $# print arg 1 is $1 print arg 2 is $2 print arg 3 is $3 50

Positional Parameters Here is an example of the use of the newpos script: $ newpos 76 trombones starting args are 76 trombones number of args is 2 arg 1 is 76 arg 2 is trombones new args are NCC 1701 Enterprise number of args is 3 arg 1 is NCC arg 2 is 1701 arg 3 is Enterprise 51

Positional Parameters • What if there are more than 10 positional parameters? $ cat ten_args print arg 10 is $10 $ ten_args a b c d e f g h i j arg 10 is a 0 • Use the shift command to perform a shift of the positional parameters n positions to the left. (Default n=1) • The shell variables $# and $* also change when the shift command is used (along with the values of the positional parameters. • Once a default shift is performed, the first parameter is discarded unless it is saved in a user-defined variable 52

Positional Parameters Examples of shift usage: $ cat shift_it print $#: $0: $* shift 3 print $#: $0 $* $ shift_it It was the best of times, it 12: shift_it: It was the best of times, 11: shift_it: was the best of times, it 10: shift_it: the best of times, it was 7: shift_it: times, it was the worst of times $ cat ten_args arg 1=$1 shift print $arg 1 $* $ ten_args 1 2 3 4 5 6 7 8 9 10 53

Referencing Variables • So far we have seen that the syntax for referencing a variable (obtaining its contents) is $var_name • An alternate syntax which has more capabilities is ${var_name} This syntax: • u Allows a script to easily use more than 10 positional parameters v Allows for more general use of a variable contents w Allows for the number of characters in the variable’s string contents to be determined x Allows for parameter substitution: replacement of the value of a variable with an different value based on specific conditions and events. 54

Referencing Variables u Greater than 10 positional parameters: $ cat moreargs #!/bin/ksh print arg 10 is print arg 11 is $10 ${10} $11 ${11} $ moreargs what rough beast, its time come at last, slouches toward Bethlehem arg 10 is what 0 arg 10 is toward arg 11 is what 1 arg 11 is Bethlehem 55

Referencing Variables v More general use $ cat copy set -x file=lone cp $filewolf cp $file ${file}wolf ls lone* $ copy + file=lone + cp lone Insufficient arguments Usage: cp [-a. Dfir. Rp] [-b size] [-t. P -[e size]] f 1 f 2 cp [-a. Dfir. Rp] [-b size] [-t. P -[e size]] f 1. . . f 2 dir + cp lonewolf + ls lonewolf w Length of a Variable’s contents $ cat length book="Pillars of the Earth" print length of book= ${#book} $ length of book= 20 56

Parameter Substitution • Misnamed: should be variable substitution • Four syntaxes for parameter substitution ${var_name: - value} ${var_name: = value} ${var_name: + value} ${var_name: ? value} • The meanings and typical usages of these four methods are shown on the following pages 57

Parameter Substitution ${parameter: -value} • If parameter exists and isn't null, return its present contents; otherwise return value. • Purpose: returning a default value if the variable is undefined. • If parameter is null, the value returned is temporary. The null value of the variable is not permanently altered • Interactive Examples: $ count="" $ print You have ${count: -5} cards You have 5 cards $ print $count $ FOO=/usr/local/bin/xx $ print edit file ${FOO: -$HOME/xx} edit file /usr/local/bin/ xx $ print $FOO /usr/local/bin/xx 58

Parameter Substitution • As the name implies, this particular syntax of parameter substitution can be used positional parameters. • Consider the following script that searches for a file: $ cat fnddemo #!/bin/ksh set -x find ${2: -$HOME} -name $1 -print $ fnddemo bk. txt. Z. + find. -name bk. txt. Z -print. /bk. txt. Z $ fnddemo dale 11. ps + find /homea/dje -name dale 11. ps -print /homea/dje/dale 11. ps 59

Parameter Substitution ${parameter: =value} • If parameter exists and isn't null, return its present contents; otherwise set it to value and then return value • Purpose: Setting a variable to a default value if it is undefined. • This form of parameter substitution cannot be used with positional parameters because their values can only be changed with the set command • Interactive Example: $ count="" $ print You have ${count: =5} cards You have 5 cards $ print $count 5 60

Parameter Substitution ${parameter: +value} • This substitution format acts in an opposite manner to the previous two. • If parameter exists and is not null, return value; otherwise return null. • Purpose: Testing for the existence of a variable • Interactive Example: $ cat trace set -x trace=$1; print ${trace: +"Trace mode on"} $ trace y + trace=y + print Trace mode on $ trace 2 + trace= + print $ 61

Parameter Substitution ${parameter: ? message} • If parameter exists and isn't null, return its contents; otherwise print scriptname: parameter: message • and abort the current script. Omitting message produces the default system message. • Purpose: Catching errors that result from variables being undefined which need to be. • Examples: $ cat fnd 2 find ${2: -$HOME} -name ${1: ? "No file specified"} $ fnd 2: 1: No file specified $ cat fnd 3 find ${2: -$HOME} -name ${1: ? } -print $ fnd 3: 1: parameter null or not set -print 62

Parameter Substitution • NOTE: In all forms, the colon (: ) is optional. If omitted, the operator tests for existence only, i. e. , change "exists and isn't null" to "exists" in each definition. 63

Advanced Pattern Matching • Korn shell enhancements to the set of pattern-matching operators: Operator Meaning If the pattern matches the beginning of the variable's value, delete the shortest part that matches and return the rest. ${variable##pattern} If the pattern matches the beginning of the variable's value, delete the longest part that matches and return the rest. ${variable%pattern} If the pattern matches the end of the variable's value, delete the shortest part that matches and return the rest. ${variable%%pattern} If the pattern matches the end of the variable's value, delete the longest part that matches and return the rest. ${variable#pattern} Mnemonic: # matches the front because number signs precede numbers; % matches the rear because percent signs follow numbers. 64

Advanced Pattern Matching • Typical use of the above pattern-matching operators is stripping off components of pathnames such as directory prefixes and filename suffixes. • Example: Assume the variable TEST has the value /home/job/src/long. file. name then: Expression ${TEST##/*/} ${TEST} ${TEST%. *} ${TEST%%. *} Result long. file. name job/src/long. file. name /home/job/src/long. file /home/job/src/long 65

Exercises 1. Write a script called lsc which executes the command ls -C. To execute this command you must give the full path name to your lsc script. Make the lsc shell script executable and run it. 2. Write a script called b which changes your current directory to /bin, displays the current directory, and then executes the lsc script created above. Make the b script executable and run it. What is your current directory when the b script is finished executing? Why? 3. Write a script called ex_on, which turn on execute permission on the first argument on the command line. 4. Modify ex_on to turn on execute permission on all arguments on the command line. 5. Write a script called 12 that prints the twelfth argument on the command line. 66

Exercises 6. Suppose you keep a file of your home coin collection that keeps track of how many coins you have in a given category. Lines in this file look like: 62 11 36 9 U. S. proofs pennies (1850 -1908) pennies (1909 -1950) nickels (1861 -1938) Write a script that prints the N types of coins of which you have the most. The default for N is 10. The program should take one argument for the name of the input file and an optional argument for how many lines to print. 67

Decision Making Commands • • • Flow Control Exit Status II if/then/else Command Condition Tests Logical Operators Negation Operator Illustrative Examples Case Statement Exercises 68

Flow Control • Flow control gives a programmer the power to specify that only certain blocks of shell commands are executed • Korn shell supports the following decision making constructs: if/else Execute a list of statements if a certain condition is/is case not true Execute one of several lists of statements depending on the value of a variable 69

Exit Status II • • For decision making, we will look at the exit status in a different way then we did before The constructs if, while, until and the logical AND (&&) and OR (||) operators use exit status to make logical decisions: A 0 value is a logical "true" (success) A nonzero is a logical "false" (failure) • There are built-in true and false commands which you can use. The true command returns an exit status of 0, the false command returns an exit status of 1. 70

Exit Status of a Script • A script, like any other process, sets an exit status when it finishes executing. Shell scripts will finish in one of the following ways: Abort - If the script aborts due to an internal error, the exit status is that of the last command (the one that aborted the script). End - If the script runs to completion, the exit status is that of the last command in the script Exit - If the script encounters and exit command, the exit status is that set by that command • Syntax for the exit command is exit [num]. When the exit command is encountered the script ends right there and the exit status is set to num 71

if / else • Simplest type of decision making construct is the conditional embodied in Korn shell's if statement. • Syntax: if condition then statements [elif condition <-- can use multiple elif clauses then statements. . . ] [else statements] fi <-- must be by itself on the final line of the construct 72

if / else • The if statement uses an exit status to determine whether or not to execute the commands. • Statements are executed only if the given condition is true. • If one or more elifs are used, the last else clause is an "if all else fails" part. 73

Condition Tests • The if construct can only test exit status but that doesn't limit you to checking only whether commands ran properly or not. • Using the [[ ]] operator, many different attributes can be tested: – string comparisons – file attributes – arithmetic conditions • [[ condition ]] just returns an exit status that tells whether condition is true or not • [[ ]] surround expressions that include various types of relational operators. 74

Condition Tests: Strings String comparison operators: Relational Operator str = pat str != pat str 1 < str 2 str 1 > str 2 -n str -z str True if. . . str matches pat. str does not match pat. str 1 is less than str 2. * str 1 is greater than str 2. * str is not null (has length greater than 0). str is null (has length 0). *based on the ASCII value of their characters str refers to an expression with a string value, and pat refers to a pattern that can contain wildcards. 75

String Comparison Examples $ cat compstr x=castle; y=cast* if [[ $x = $y ]] then print $x matches $y else print $x does not match $y fi $ compstr castle matches cast* $ cat compstr 1 x=hello; y=helper if [[ $x > $y ]] then print "$x is > $y" else print "$x is not > $y" fi $ compstr 1 hello is not > helper 76

String Comparison Examples $ cat compstr 2 if [[ -n $1 ]] then print First argument is $1 fi if [[ -z $1 ]] then print Where is the first argument? fi $ compstr 2 Logan First argument is Logan $ compstr 2 Where is the first argument? 77

Condition Tests: Files File attribute checking: Relational Operator -a file -d file -f file -r file -s file -w file -x file -O file -G file 1 -nt file 2 file 1 -ot file 2 True if. . . file exists file is a directory file is a regular file there is read permission on file is not empty you have write permission on file you have execute permission on file, or directory search permission if it is a directory you own file your group id is same as file's file 1 is newer than file 2 file 1 is older than file 2 There are 21 such operators in total. 78

File Checking Demo Directory • Consider a directory that contains the following files and subdirectories $ls -lrt -rwxr--r--rw-r--r-drwxrwsr-x -rw-rw-r-drwxr-sr-x -rw-r--r--rw-r--r--rwxr-xr-x -rw-r--r-- 1 1 1 2 2 1 1 1 dje dje dje dje appl appl appl appl 5 512 0 0 0 80 80 0 Jun Jun Jul Sep May May 27 27 27 3 4 18 18 2000 2000 11: 54 11: 55 11: 57 12: 01 12: 04 file my_prog my_scr. 1 scripts slide 1. ps probs solutions court. msg heel greed file. c file. f 90 typescript 79

File Checking Examples $ cat fexist if [[ -a $1 ]] then print File: $1 does exist else print File: $1 does not exist fi $ fexist sort. c File: sort. c does not exist $ cat ffull if [[ -s $1 ]] then print $1 has something in it else print $1 is empty fi $ ffull heel is empty $ cat fx if [[ -x $1 ]] then print I can execute $1 else print I cannot execute $1 fi $ fx file I can execute file 80

Condition Tests: Integers • Need to use a new test operator for integer comparisons (( … )) instead of [[ … ]] Relational Operator < > <= >= == != Meaning less than greater than less than or equal greater than or equal not equal 81

Integer Comparison Examples $ cat intcomp if (( $1 < $2 )) then print $1 is less than $2 fi if (( $1 <= $2 )) then print $1 is less than or equal to $2 fi if (( $1 == $2 )) then print $1 is equal to $2 fi if (( $1 > $2 )) then print $1 is greater than $2 fi if (( $1 >= $2 )) then print $1 is greater than or equal to $2 fi if (( $1 != $2 )) then print $1 not equal to $2 fi 82

Integer Comparison Examples $ 3 3 3 intcomp 3 89 is less than or equal to 89 not equal to 89 $ intcomp 13 13 13 is less than or equal to 13 13 is greater than or equal to 13 83

Logical Operators • Two special symbols allow the user to combine relational tests into more complex conditions && is a logical AND || is a logical OR • How they work: statement 1 && statement 2 will execute only if statement 1 was successful (returned an exit status of 0) statement 1 || statement 2 will execute only if statement 1 was not successful (returned a nonzero exit status). 84

Logical Operators • The logical operators && and || are commonly used with if/else commands • Examples: $ ls -l -r-xr-xr-x 1 dje appl 365 May 18 -rw-r--r-1 dje appl 209 May 18 $ cat setw #!/bin/ksh if [[ -f $1 && -x $1 ]] then print You are getting write permission for $1 chmod u+w $1 fi $ setw length. txt $ setw intcomp You are getting write permission for intcomp $ ls -l -rwxr-xr-x 1 dje appl 365 May 18 -rw-r--r-1 dje appl 209 May 18 15: 13 intcomp 15: 12 length. txt 85

Logical Operators $ cat gone #!/bin/ksh if [[ $1 = quit || $1 = bye ]] then print You have closed correctly else print Improper closing word exit 11 fi $ gone bye You have closed correctly $ echo $? 0 $ gone quit You have closed correctly $ echo $? 0 $ gone end Improper closing word $ echo $? 11 86

The Negation Operator • If an exclamation point ! is put before any of the conditions discussed, it reverses the “truth or falseness” of the condition. • [[ ! -d probs ]] returns true (exit status 0) if probs is not a directory • [[ ! Bob = ? o? ]] returns false (exit status 1) even though Bob matches the pattern 87

Combining Different Types of Conditions • If you want to combine an integer comparison with file or string comparison within the same condition, you must do two things: – Use [[ … ]] for the test operator – Use the integer relational operators shown below Relational Operator -lt -le -eq -ge -gt -ne Comparison less than or equal greater than not equal 88

Combining Conditionals Example $ cat fileint #!/bin/ksh let num=$2 if [[ $1 < tact || num -ge 3 ]]; then print Pippin else print Merry fi $ fileint ace 45 Pippin $ fileint vice 2 Merry 89

Illustrative Examples Test operator run interactively $ $ 0 $ $ 1 [[ -z "" ]] print $? [[ -z foo ]] print $? Treating integers strings as strings: compressed format $ if [[ 6 > 57 ]] ; then print huh? ; fi huh? 90

Illustrative Examples The following script sets user execute permission on an ordinary, file that is not a directory: if [[ -f $1 && ! -d $1 ]]; then chmod u+x $1 fi 91

Illustrative Examples The following script removes the first file if it's older than the second file and the variable KEY is non-null: if [[ $1 -ot $2 && -n $KEY ]] then /bin/rm $1 fi 92

Illustrative Examples The following script compares two files and if no differences are found, removes the second file. Also helpful USAGE programming style: USAGE="Usage: t $0 file 1 file 2" if [[ $# -ne 2 ]]; then print ${USAGE} exit 1 fi diff $1 $2 > /dev/null if [[ $? -eq 0 ]]; then /bin/rm $2 print $2 removed, $1 kept else print $1 and $2 differ fi 93

Case command • Provides a multiple choice decision structure. When executed one of several statement blocks will be run. • Replacement for the “elif-ladder” • Lets you test strings against patterns that can contain wildcard characters. • Syntax: case expression in pattern 1) statements ; ; pattern 2) statements ; ; . . . esac 94

Case Operation • If expression matches one of the patterns, its corresponding statements are executed • If there are several patterns separated by pipes, the expression can match any of them in order for the associated statements to be run ( | acts as logical or ) • are checked in order for a match; if none is found, nothing happens Patterns • When the double colons (; ; ) are encountered at the end of the one of the statement lists, the case terminates 95

Case Examples Here's a simple script which moves C and Fortran source files to one directory and object code files to another: case $file in *. c|*. f) /bin/mv $file ${HOME}/src ; ; *. o ) /bin/mv $file ${HOME}/obj; ; * ) print $file not moved ; ; (Default value or USAGE) esac (Could have also used *. [cf] wildcard construct above) 96

Case Examples • The case statement is often used to specify options for a shell script. • Here is a shell script called dt_fmat that allows the user to enter options that affect the way the date is displayed: case $1 in -d) print -n Date: date +"%a %h %d" ; ; -t) print -n Time: date +"%T" ; ; -w) print -n Weekday: date +"%a" ; ; -y) print -n Year: 20 date +"%y" ; ; -m) print -n Month: date +"%h" ; ; *) print -n Date: date +"%a %h %d" print -n Time: date +"%T" ; ; esac 97

Case Usage $ date Sat May 19 12: 01: 57 EDT 2001 $ dt_fmt -d Date: Sat May 19 $ dt_fmt -t Time: 12: 05 $ dt_fmt -w Weekday: Sat $ dt_fmt -y Year: 2001 $ dt_fmt -m Month: May $ dt_fmt Date: Sat May 19 Time: 12: 02: 54 98

“Elif-Ladder” vs Case • The following script shows and “elif-ladder” and a case statement that do the exact same thing. Note how the case statement has a simpler syntax if [[ $1 = cat || $1 = dog ]]; then print You have a normal pet elif [[ -z $1 ]]; then print You have no pets else print You have a wierd pet fi case $1 in cat|dog) print You have a normal pet ; ; '') print You have no pets ; ; *) print You have a wierd pet ; ; esac • Advantage of “elif-ladder”: not limited to string comparisons 99

Exercises 1. Write a script called char that checks a single character on the command line, c. If the character is a digit, digit is displayed. If the character is an upper or lowercase alphabetic character, letter is displayed. Otherwise, other is displayed. Have the script print an error message if the argument c is more than one character in length. 2. Write a script called mycopy that copies a source file to a directory. Add a check to see if the source file exists. If the source file does not exist, print an error message. 3. Write a script called mydir that prints the message File is a directory if the file is a directory. 100

Exercises 4. Write a script called ver that accepts one argument on the command line indicating the name of a file. The file is copied to another file with the same name with the addition of the extension. v 2. Also, the line #Version 2 is added to the top of the file. 5. Execute the ver script on itself, creating a new version of the file called ver. v 2. 6. Rewrite ver. v 2 to accept a possible second argument. If two arguments are entered, the file specified by the first argument is copied to a file with the name of the second argument. If no second argument is entered, the file is copied to another file with the same name, adding the extension. v 2. In either case, the line #Version 2 is added to the top of the file. 101

Looping Commands • • • for select while until Exercises 102

for Loop Command • • • The for loop allows a section of commands to be run a fixed number of times During each time through an iteration, the loop variable is set to a different value. Syntax: for name [in list] do statements that can use $name done where — name is the loop variable which can be called anything — list is a list of strings (defaults to $*) separated by spaces • Operation: name is set to each string in list, in order, for each iteration; the number of iterations equals the number of names in list. 103

for Loop Examples Outputting a Phrase $ cat simple for word in This is a test do print $word done $ simple This is a test <-- Omitting the in This is a test phrase and instead running the script as $ simple This is a test would yield the same output. 104

for Loop Examples The list can be the contents of a local variable: Check to see who is logged on the machines listed in the variable SYSTEMS="myrtle gull sandy newport daytona": for sys in $SYSTEMS do finger @$sys print done 105

for Loop Examples list can contain shell wildcards and command substitution as well: $ cat file 1 for i in * do print $i done $ file 1 file 2 $ cat file 2 for i in $(ls) do print $i done $ file 2 file 1 file 2 106

for Loop Examples • Can explicitly make the list all the command line arguments (also notice the alternate syntax): $ cat upper #!/bin/ksh for bug in $* ; do typeset -u day=$bug print $day done $ upper Robson Bond Green ROBSON BOND GREEN 107

select Command • select allows you to generate simple interactive ASCII menus easily. • Syntax: select name [in list] do statements that can use $name done • This is the same syntax as the for loop except for the keyword select. As with for, in list defaults to $* if omitted. 108

select Operation • Each item in list is displayed as the choice in the menu with a number preceding it. At the bottom of the menu the PS 3 prompt is displayed and script is waiting for the user to select a choice by entering its number. (A default value for the PS 3 prompt is provided by the system but the user can change it) • The selected choice is stored in name and the selected number in a system-predefined variable called REPLY • The statements in the body of the do loop are run • Repeats the process forever; exit loop with break statement (or user can issue ctrl-d) 109

select Example The following script termselect allows you to select a terminal setting: PS 3='terminal? ' select term in vt 100 vt 220 xterm do if [[ -n $term ]]; then TERM=$term print TERM is $term else print invalid choice fi done 110

select Example $ termselect 1) vt 100 2) vt 220 3) xterminal? 4 invalid choice terminal? 3 TERM is xterm 111

select Example $ cat travel select i in Past Present Future; do print You have picked choice $REPLY print "You will go to the $i" break done $ travel 1) Past 2) Present 3) Future #? 1 You have picked choice 1 You will go to the Past $ travel 1) Past 2) Present 3) Future #? 3 You have picked choice 3 You will go to the Future $ 112

while Command • • Allows a section of code to be run repetitively while a certain condition holds true. Syntax: while condition do statements. . . done • Operation: while condition remains true (exit status 0) the body of the while loop is run • [[ ]]operator can be used here as with if to specify the condition • • Beware of creating an infinite loop (condition must become false at some point). If condition is false when a while loop is first encountered, the loop will not be executed even once 113

while Example: print out positional parameters: $ cat args while [[ $# -ne 0 ]]; do print $1 shift done $ args There are 7 “deadly sins” There are 7 deadly sins 114

while Example: Using a while as a counting loop: $ cat whileint print $#: $3 let i=1 while (( i <= 5 )); do shift let i=i+1 done print $#: $3 $ whileint 1 3 5 7 9 11 13 15 17 19 10: 5 5: 15 115

until Command • • Allows a section of code to be run repetitively until a certain condition becomes true. Syntax: until condition do statements. . . done • Operation: while condition remains false (exit status nonzero) the body of the until loop is run • [[ ]]operator can be used here as with if to specify the condition • • Typically, condition is false when an until loop is first encountered; thus the loop will not be executed at least once Just about any until can be converted to a while by simply negating the condition 116

until Example • Classic example: checking user input $ cat prompt #!/bin/ksh until [[ $ans == no || print -n 'Continue? read ans done $ prompt Continue? (yes or no): Continue? (yes or no): $ $ans == yes ]]; do (yes or no): ' y sure ok yes n nope no way no 117

Exercises 1. Write a script called lis that uses a for loop to display all files and directories in the current directory. 2. Write a script called dirs that displays only the names of the directory files in your current directory. 3. Using the select command, write a script that creates a menu of the abbreviations for all the months of the year. If a certain month abbreviation is chosen, the script should print out the full name of the month and number of days in it. Change the PS 3 prompt to something appropriate. 4. Using a while loop, write a script that displays a guessing game. The player is to guess a number between 0 and 59. If the guess is too low, the message “Higher” is printed. If the guess is too high the message “Lower” is printed. Guesses continues in until the player guesses the correct number. The number the user is trying to guess is taken from the seconds of the date command. 118

Exercises 5. Modify the script from problem 4 to keep track of the number of guesses it takes for the user to discover the correct number. Have the script print the total number of guesses that it took. 6. Rewrite the first while example script shown in this chapter (printing out the positional parameters) using the until statement instead. 119

Working with Integer Variables • • • Arithmetic Operations Operator Precedence Numerical Bases String Variables vs Integer Variables Exercises 120

Arithmetic Operations • • So far we have seen how to give values to integer variables and compare them (in two different ways!) In this section, integers will be used in arithmetic operations Below is a list of the most common operations performed with integers Arithmetic Operator + - addition * multiplication / division with truncation modulus division % • • Meaning subtraction The shell interprets symbols surrounded by $(( and )) as arithmetic expressions. Variables in arithmetic expressions do not need to be preceded by dollar signs. • 121

Arithmetic Examples 1. Demonstration of the integer divisions: 2. 3. 4. 5. 6. 7. 8. 9. $ cat intop let x=31 let y=8 print The integer division print The modulus division $ intop The integer division of 31 The modulus division of 31 of $x by $y is $((x/y)) of $x by $y is $((x%y)) by 8 is 3 by 8 is 7 10. Parentheses can give operations higher precedence: 11. $ cat paren 12. #!/bin/ksh 13. let x=$(( $1 + $2 * $3 - $4 )) 14. print $x 15. let x=$(( ($1 + $2) * ($3 - $4) )) 16. print $x 17. $ paren 2 4 6 3 18. 23 19. 18 122

Different Numerical Bases Possible to represent integers in alternate bases: $ cat base #!/bin/ksh print -n $(( print -n , print -n $(( print $ base 57, 57, 57 $ 57 )) 16#39)) 8#071)) 2#00111001)) 123

String Variables vs Integer Variables Arithmetic does not work with string variables: $ cat intstr print As integers. . . if (( $1 < $2 )) then print $1 is less than $2 fi let x=$2+4; print $2+4 is $x print As strings. . . if [[ $1 > $2 ]] then print $1 is greater than $2 fi x=$2+4; print $2+4 is $x $ intstr 3 15 As integers. . . 3 is less than 15 15+4 is 19 As strings. . . 3 is greater than 15 15+4 is 15+4 124

Exercises 1. Write a script which finds the sum of all the positional parameters. 2. Write a script to determine if the first positional parameter is prime. 3. Write a script to perform the factorial of the first positional parameter 4. Write a script that will output positive and negative integers. 5. Write a script that will convert the first positional parameter from base 10 to base 5. 6. Write a script that reads in a time in seconds and outputs as hours: minutes: sec. 125

Script Input / Output • • • Input / Output Redirection “Here-is” documents print read Quoting Exercises 126

Input / Output Redirection • The basic I/O redirectors >, < and | are good enough for most tasks. • The Korn shell supports 16 redirectors in total. • Here are some other useful ones: Redirector Function >> file >| file Direct stdout to file, appending to file if it already exists force stdout to file even if noclobber is set << label n> file n<file >&n <&n “here-is” document (see below) direct file descriptor n to file set file as file descriptor n duplicate stdout to file descriptor n duplicate stdin from file descriptor n 127

Input / Output Redirection Example: This script called start which can be used to start a different script (entered as positional parameters) in the background and save both stdout and stderr in a single log file. $* > logfile 2>&1 & If we want to do the same as above but to the screen as well: $* 2>&1 | tee logfile & The command tee takes its stdin and copies it to stdout and to the file given as argument. 128

“Here-is” documents • Special form of inline input redirection • Syntax: command << label input for command. . label • The shell performs command variable substitution where appropriate unless label is surrounded by quotes 129

print • We have already seen many examples of the print command. • Here are some useful escape sequences and options; enclose escape sequences in quotes (either double or single): Escape Sequence a b r t Character printed bell backspace return tab Option -n -r -s -un Function omit the final newline ignore any escape sequence print to history file print to file descriptor n 130

print Example $ cat escape #!/bin/ksh print "Call me t Ishmael" print -r "Call me t Ishmael" $ escape Call me Ishmael Call me t Ishmael $ cat print. u 3 #!/bin/ksh print -u 3 "My errors: " 3>errors $ ls errors Cannot access errors: No such file or directory $ print. u 3 $ cat errors My errors: 131

read • We have seen simple examples of read. • Shell scripts with read are useful for certain tasks such as reading in lines from a file • Suppose we have the file /etc/nicks : myrtle killian daytona. . . beach_bum brewster speedster which is a list of hostnames followed by nicknames. We want to assign the variable NICKNAME to the appropriate nickname depending on which machine you log into. 132

read • The following code will do it: machine=$(uname -n) while read host nick; do if [[ $host = $machine ]]; then NICKNAME=$nick print NICKNAME is $NICKNAME break fi done < /etc/nicks <-- read exits with status 1 when there is nothing left to read 133

Quoting • A source of confusion for novice and experienced Korn shell script writers is when to use single quotes ‘ ‘ and when to use double quotes “ “ • Throughout the workshop we have learned that certain symbols have special meanings: $name indicates the contents of the variable name, $(command) indicates the output of a command, etc. • Single quotes are strong quotes: they prevent the special meaning symbols from performing their tasks. The symbols are treated just like normal characters. • Double quotes are weak quotes: they allow the following actions to take place: – variable and positional parameter substitution – command substitution – arithmetic calculations 134

Quoting Demonstration $ cat quote #!/bin/ksh money=franc print "Nonsense words ${money} $1 $(date +%T) $((3*75))" print 'Nonsense words ${money} $1 $(date +%T) $((3*75))' $ quote raven Nonsense words franc raven 16: 23: 30 225 Nonsense words ${money} $1 $(date +%T) $((3*75)) 135

Exercises 1. Write a shell script tfile which creates a test file called junk. Do not use a cp or mv command to accomplish this. File junk contains the following: This is a test abcdefghijklmnopqrstuvwxyz 0123456789 end of test 2. Create three files called data 0, data 1, and data 2 which contain the month, day, and year respectively of your birthday. Write a single script that reads from all three of these files and then outputs your birthday to the screen 136

Communicating with Running Scripts • • Sending Signals Signal Handling 137

Signal Handling • We want to have control over our shell scripts even when they are interrupted by signals. • Signals come in the form of control keys, the kill command user-initiated events like logging out. • Programs can be set up to "trap" specific signals and process them in their own way. • Syntax of trap command: trap command sig 1 sig 2. . . when any of sig 1, sig 2, etc. , are received, run command, then resume execution • 138

Signal Handling • After command finishes, the script resumes execution just after the command that was interrupted. • sig 1, sig 2, . . . can be given by name or number (name is more portable). • trap without arguments lists traps that have been set 139

Signal Handling Simple example - shell script loop : trap 'print You hit ctrl-c!' INT while true: do sleep 60 done ctrl-c will not kill this program but rather print the message and start another sleep; type ctrl- to stop it (then rm core). 140

Signal Handling Enhancing the script to handle background execution (trap the kill command): trap 'print while true: sleep done You hit ctrl-c!' INT No kill I!' TERM do 60 if run as loop &, kill %loop won't stop it; need kill -KILL %loop Note: kill -l will give the numbers associated with given signals (defined in /usr/include/sys/signal. h). 141

Signal Handling Examples: • If any of the signals listed (0, 1, 2, 3, or 15) are received by the program, file temp$$ will automatically be removed. Without the exit command, program execution would continue from the point where the signal was received: trap 'rm -f temp$$; exit 0' 0 1 2 3 15 • The command listed for trap is null. This specifies that the signals listed are to be ignored, and the program will try to continue where it left off: trap "" 0 1 2 3 15 • Using the null command (: ) specifies that the signals listed are to be ignored for the next command only: trap : 0 1 2 3 142

Signal Handling • What if you want a program running in the background to continue even after you log out? Shell sends the HUP (hangup) signal to all of your background jobs when you log out. • Use the command nohup to put a job into the background which will ignore HUP and QUIT (ctrl-) signals: nohup $* > logfile 2>&1 & 143

Signal Handling Most common signals: Signal 1 2 3 9 15 24 Name HUP INT QUIT KILL TERM TSTP Note Hangup ctrl-c ctrl- Cannot be trapped default kill signal ctrl-z 144

Script Functions • • • Why Use Functions? Syntax & Use Auto-Loading Function Definition Location Command Precedence Exercises 145

Functions • Improves shell's programmability – already in memory (unless autoloaded) – modular programing • Syntax: function functname { shell commands } or, functname () { shell commands } 146

Functions • Delete a function definition with $ unset -f functname • Display all defined functions: $ functions 147

Functions • Two important differences between functions and shell scripts run by name (Method 2): – functions do not run in a subprocess; behave more like a script run by Method 1 – functions have precedence over scripts • Where to put them? – enter on command line (no longer available if you log out and log back in) –. profile (only available in login shell) – $ENV file (available to all child shells) – in $FPATH directory (Korn shell will use autoload feature) • Function names should begin with a letter or underscore • The closing brace (}) must appear on a line by itself 148

Functions Example: Run a program and view the output $ > > > > myrun () { cd $WORKDIR a. out < data. in more data. out /bin/rm data. in mv data. out data. in } 149

Exercises 1. Write a script which sets the PAGER environment variable. The script should accept input values from: • command-line arguments: • -p pager or -ppager • shell environment variable PAGER • default setting: more -s Include a usage line. (Hint: You should try using the pattern matching operators introduced in this section) 2. Write a script that implements a simple version of the shell's built-in whence command. It should check each of the directories on your PATH for a command given as an argument, and print the full pathname if the command is found. Only files that are executable by you (and are not directories!) should be returned. 150

Exercises 3. Design functions pushd and popd which implement a "stack" of directories (as in the C shell). • The stack is just a list of directories, stored in an environment variable we'll call DIRSTACK. • pushd should "push" a specified directory onto the stack, i. e. , prepend the directory name to DIRSTACK. • popd "pops" a directory off the stack, i. e. , cds to the top directory on the stack and then removes that directory from the stack. • Your functions should: – Allow the use of ". " to specify the current directory. – Return an error message if pushd is invoked without an argument, or if the specified directory does not exist. – popd should return an appropriate message and do nothing if the stack is empty. • • Why are these implemented as functions rather than scripts? What modifications (if any) should you make to your ~/. profile in order to use them? 151

Exercises 4. Write a script that will change the suffix of all the files in the present directory from. old to. new 152

HPC Designed Scripts • • Repeating Program Execution Debugging Options Batch Jobs Performance Tool Preprocessing Multiple Tool Runs Multiple Tool Reports File Renaming 153

Repeating Program Execution • • It is often desired to run a program several times. Even a serial programs may have to run repeatedly to debug intermittent problems, find logical errors, etc. In HPC parallel processing arena, there are even more reasons to write a script that will execute code many times: – Each run is with a different number of processors so that the traditional speedup graph data can be collected (and compared with Ahmdal’s Law) – To perform repeated timing of the entire or partial code. • • • Several parallel machines do not allow dedicated use of processors A severe load imbalance may come and go depending on how concurrently the various processors actually work Over synchronized program’s times can vary due to the order of arrival of the processors at barriers – To gather data from program output for statistical analysis. One can also gather timing data for similar reasons. – To confirm that a parallel program is preventing race conditions which can arise when one processor is interrupted by another when the former is trying to read, update, and write back to a shared memory location • Repeated runs is easily done in our introductory triangle script by using a while loop. Here the script will run the timearea code which times how long the program takes to calculate the area 154

Timing Script $ cat triangle cc -n 32 -cckr -O 3 -LNO -Ofast -TARG: madd=on -o $1 $1. c cat << EOF > sides. in 8 3 7 EOF let i=0 while (( i < $2 )); do $1 < sides. in >> $1. dat let i=i+1 done cat $1. dat rm $1 sides. in 155

Timing Script Usage & Output $ triangle timearea 17 1. 215377 1. 227504 1. 197454 1. 193690 1. 220033 1. 190733 1. 194406 1. 221519 1. 201336 1. 187614 1. 197026 1. 220640 1. 217389 1. 218919 1. 192417 1. 218107 1. 228289 156

Debugging Options • To use debugging software (dbx, Totalview, Xbrowse, etc) programs need to be compiled with certain options. • The debugging option is traditionally –g but has been improved lately to include various levels of optimizations while the debugging is being performed (G 0, G 1, …) • For all debugging options, the compiler will make a symbol table which connects a program identifier (variable name, etc. ) with a memory address. This allows the debugger to track the values of any variables as the program is executed line by line • On next page we show a variation of the introductory triangle script in which a decision is made- through the use of a second positional parameter- if debugging will be implemented 157

Debugging Script 1. $ cat triangle 2. if [[ $2 = debug ]]; then 3. cc -g -o $1 $1. c 4. print "$1 ready to be debugged" 5. else 6. cc -n 32 -cckr -O 3 -LNO -Ofast -TARG: madd=on -o $1 $1. c 7. $1 < sides. in > $1. dat 8. cat $1. dat 9. rm $1 10. fi 11. $ triangle area 12. Area of Triangle is 10. 392305 13. $ triangle area debug 14. area ready to be debugged 158

Batch Jobs • There a number of popular batch processing software that are heavily used on HPC machines: NQE, PBS, and LSF to name three. • At this point in the course, we now realize that the “batch files” we are used to writing are actually just shell scripts. The header lines have the syntax and are treated as comments if the batch script is not submitted to the batch software. • But since these batch scripts have designed purpose – to run the UNIX commands in them and send a log file backwe can tailor them to reflect it. • For example, it is recommended that the shell command echoing feature be turned on so that the commands appear in the log file preceding their output. An example is shown on the next slides. 159

A Batch Script • Here is a simple batch job (file sp_gravity. job): #QSUB -r grav #QSUB –l. T 3000 #QSUB –l. M 10 Mw #QSUB -oe set -x # echo commands cd gravity/batch ls -lrt f 77 -o spg sp_gravity. f. /spg < region. rsp ls –lrt 160

A Log file (grav. o 3613) + ls -lrt total 1512 -rw-r----1 dje G-541 46098 May 18 12: 22 sp_gravity. f -rw-r----1 dje G-541 637236 May 18 12: 22 newbedrx. cray -rwxr----1 dje G-541 299 May 18 12: 31 region. prm -rw-r--r-1 dje G-541 16 May 18 12: 31 region. rsp -rw-r--r-1 dje G-541 125 May 18 12: 55 sp_gravity. job + f 77 -o spg sp_gravity. f: MAIN xy 2 glqr: "sp_gravity. f", line 346: Warning: local variable "zero" never used gauleg: +. /spg + 0< region. rsp Parameter file: Gravity output file: region. out 3 -D ORTHOGONAL COLLOCATION ON FINITE ELEMENTS ***** (GAUSS-LEGENDRE QUADRATURE). . . Integration over source region in progress. . . 161

Log file (cont. ) + ls -lrt total 1712 -rw-r-----rwxr-----rw-r--r--rwxr-xr-x -rw-r--r-- 1 1 1 1 dje dje G-541 G-541 46098 637236 299 16 125 67228 2128 14502 May May 18 18 12: 22 12: 31 12: 55 12: 58 13: 00 sp_gravity. f newbedrx. cray region. prm region. rsp sp_gravity. job spg region. coeff region. out 162

qsub Command with options • The realization that a batch job is really just a normal shell script, is clear when the script only consists of the Unix commands. • The “header” section is replaced by options to the qsub command: qsub –r grav –l. T 3000 –l. M 10 Mw –eo < grav. ksh • Where grav. ksh is the remaining Unix commands: set -x cd gravity/batch ls -lrt f 77 -o spg sp_gravity. f. /spg < region. rsp ls –lrt • This is the alternative ( not recommended) method for running a batch job. It will return the same log file shown on the previous slides. 163

Advanced Batch Scripting • Since a batch script is – just that – a script, we can using any part of the scripting language in it. • Consider our “classic” compile and run script we have been using in the previous slides. If the compilation is a failure, there is no point in proceeding with the script’s commands since there will be no executable to work with. • In the advanced batch script shown on the next slide we use the if/then shell command to check to see if there were no errors produced in the compilation. If there was, we exit the script (and the job, BTW). • The log file returned after submitting this script is also shown. An error was deliberately inserted in the source code. 164

Error-Checking Batch Script set -x cd SV 1/dje/gravity f 90 -o spg sp_gravity. f if [[ $? -ne 0 ]]; then exit 33 fi. /spg < region. rsp 165

Error-Checking Log File + cd SV 1/dje/gravity + f 90 -o spg sp_gravity. f do i=1, ^ f 90 -197 f 90: ERROR XY 2 GLQR, File = sp_gravity. f, Line = 408, Column = 14 Unexpected syntax: "operand" was expected but found "EOS". Cray Fortran: Version 3. 5. 0. 4 (u 15 f 51 p 42138 g 41109 a 18) Cray Fortran: Mon Jun 10, 2002 17: 13: 34 Cray Fortran: Compile time: 0. 3487 seconds Cray Fortran: 1130 source lines Cray Fortran: 1 errors, 0 warnings, 0 other messages, 0 ansi Cray Fortran: "explain f 90 -message number" gives more information about each message + [[ 1 -ne 0 ]] + exit 33 166

Performance Tool Preprocessing • Almost all of the sophisticated software tools that exist on HPC machines to profile your code and aid in serial, vector, and parallel optimization require the user to run the code and generate a raw data file. The tool subsequently analyzes the raw data under the user control (usually interactively) • For some HPC software tools the time to gather the raw data can be long, especially if the code is complex with many functions and subroutines. Or the syntax required might be different than ever other optimization tool on that machine. • Because of these reasons, it is often useful to perform the generation of raw data as a shell script. • The script shown on the next slide, generates the raw data file for the prof tool on the Cray SV 1. 167

Using the prof preprocessing script oscb: $ cat profraw. ksh #! /bin/ksh f 90 –G 1 –o $1 –lprof $1. f. /$1 < $1. in # Creates the prof. data file prof –x –m prof. data $1 > $1. prof. raw # User can pick the raw output filename rm $1 oscb: $ profraw. ksh poisson oscb: $ profview poisson. prof. raw 168

Multiple Tool Runs • • For HPC tools that access Hardware Counters on the actual vectorizing processors or commodity Microprocessors (Pentiums, Itantiums, Ultra-Sparcs, MIPS R 12 K, etc. ) there are too many counter chip to gather all the information with just one run of an optimization tool. Therefore, to access all the hardware use information, your code must be run several times. For Cray machines, 4 runs of hpm and perf are required. For Microprocessor-based MPP machines there a typically 64 counters only two of which can be read out in a single run of the appropriate tool (ssrun for the Origin 2 K, collect for the Sun Fire 6800, etc. ) Compiling and running an executable, with different options chosen to access various counters is an ideal task for shell programming. In the next slide, a script is shown that gathers all the hardware use info for Cray machines with the hpm and perf tools installed 169

total_hw. ksh #! /bin/ksh # First parameter is the source code “prefix” cc –O 4 -TARG: madd=on -o $1 $1. c hpm –g 0. /$1 > $1. hpm –g 1. /$1 >> $1. hpm –g 2. /$1 >> $1. hpm –g 3. /$1 >> $1. hpm # Recompile with perf options cc –F –lperf –o $1 –O 4 –TARG: madd=on $1. c typeset –x PERF_GROUP=0 PERF_DATA=p 0. raw. /$1 typeset –x PERF_GROUP=1 PERF_DATA=p 1. raw. /$1 typeset –x PERF_GROUP=2 PERF_DATA=p 2. raw. /$1 typeset –x PERF_GROUP=3 PERF_DATA=p 3. raw. /$1 cat p 0. raw p 1. raw p 2. raw p 3. raw >perf. data rm $1 170

Saving Multiple Tool Reports • A powerful feature of HPC Optimization software tools is that they provide information on a variety of performance topics with just one run of the tool. A certain tool might have a report of subroutine time-profiling, a caller-callee tree, a summary report, etc. • When interactively using such a tool, a user can pick which report to read, and then pick another and so on. • It is often desirable to just save a select set of the possible reports. This can be done by running the tool several times but with a different option each time. The option determines what report will be saved in that run. Again, a natural job for a shell script. • In the next slide, a script is shown that saves several reports from the flowview tool used on Cray PVP machines. (12 reports are available from this tool) 171

flowview report selection script oscb: $ cat flowchoice. ksh #! /bin/ksh # Script assumes raw data file flow. data exists # First parameter is prefix of the source code pre=$1 shift for opt in $x do flowview –L$opt >> pre. flow done oscb$ flowchoice. ksh neuralnet m h u k 172

Renaming Files • Specifically, renaming the prefixes or suffixes of a large number of files. Many HPC parallel programs will produce hundreds of files which contain data at certain time steps, or spatial extent. Data visualization files are one example of this type of output. • Say the user needs to put a new suffix or prefix on a large set of files in order that they not be overwritten by a subsequent data set. • After this course, one’s mind should naturally think “ I bet a script could do this for me”. You would win that bet. • On the next slide is shown a script that will change the suffix of all files whose name has the syntax strain 001. txt, strain 002. txt to strain 500. txt to the same prefix but with “txt” replaced by “old”. The script can easily be made more general purpose as well. 173

Renaming script oscb$ cat suffix. ksh #set –x for file in $* do mv $file ${file%. *}. old done oscb$ suffix. ksh strain*. txt 174