What is Awk n n n Awk is

What is Awk n n n Awk is a programming language used for manipulating data and generating reports The data may come from standard input, one or more files, or as output from a process Awk scans a file (or input) line by line, from the first to the last line, searching for lines that match a specified pattern and performing selected actions (enclosed in curly braces) on those lines. If there is a pattern with no specific action, all lines that match the pattern are displayed; If there is an action with no pattern, all input lines specified by the action are executed upon.

Which Awk n The command is awk if using the old version, nawk if using the new version, and gawk is using the gnu version

Awk’s format n An awk program consists of: n the awk command n n If an input file is not specified, input comes from standard input (stdin), the keyboard Awk instructions consists of n n the program instructions enclosed in quotes (or a file) , and the name of the input file patterns, actions, or a combination of patterns and actions A pattern is a statement consisting of an expression of some type

Awk’s format (continue. ) n n Actions consist of one or more statements separated by semicolons or new lines and enclosed in curly braces Patterns cannot be enclosed in curly braces, and consist of regular expressions enclosed in forward slashes or expressions consisting of one or more of the many operators provided by awk commands can be typed at the command line or in awk script files The input lines can come from files, pipes, or standard input

Awk’s format (continue. ) n Format: nawk 'pattern' filename nawk '{action}' filename nawk 'pattern {action}' filename

Input from Files n Example 1: $ cat employees Chen Cho Tom Billy Larry White Bill Clinton Steve Ann 5/19/63 4/12/45 11/2/54 1/14/60 9/15/71 203 -344 -1234 913 -972 -4536 908 -657 -2389 654 -576 -4114 202 -545 -8899 $76 $102 $54 $201 $58 $ nawk '/Tom/' employees Tom Billy $ 4/12/45 913 -972 -4536 $102

Input from Files n (continue. ) Example 2: $ cat employees Chen Cho Tom Billy Larry White Bill Clinton Steve Ann 5/19/63 4/12/45 11/2/54 1/14/60 9/15/71 203 -344 -1234 913 -972 -4536 908 -657 -2389 654 -576 -4114 202 -545 -8899 $ nawk '{print $1}' employees Chen Tom Larry Bill Steve $76 $102 $54 $201 $58

Awk’s format (continue. ) n Example 3: $ cat employees Chen Cho Tom Billy Larry White Bill Clinton Steve Ann 5/19/63 4/12/45 11/2/54 1/14/60 9/15/71 203 -344 -1234 913 -972 -4536 908 -657 -2389 654 -576 -4114 202 -545 -8899 $76 $102 $54 $201 $58 $ nawk '/Steve/{print $1, $2}' employees Steve Ann

The print function n The default action is to print the lines that are matched to the screen The print function can also be explicitly used in the action part of awk as {print} The print function accepts arguments as n n n variables, computed values, or string constants String must be enclosed in double quotes Commas are used to separate the arguments: if commas are not provided, the arguments are concatenated together

The print function (continue. ) n n The comma evaluates to the value of the output field separator (OFS), which is by default a space The output of the print function can be redirected or piped to another program, and another program can be piped to awk for printing

The print function (continue. ) n Example: $ date Fri Feb 9 07: 49: 28 EST 2001 $ date | nawk '{ print "Month: " $2 "n. Year: ", $6}' Month: Feb Year: 2001

Escape sequences n Escape sequences are represented by a backslash and a letter or number Escape sequence Meaning b Backspace f Form feed n Newline r Carriage return t Tab 47 Octal value 47, a single quote c c represents any other character, e. g. , "

Escape sequences n (continue. ) Example: $ cat employees Chen Cho Tom Billy Larry White Bill Clinton Steve Ann 5/19/63 4/12/45 11/2/54 1/14/60 9/15/71 203 -344 -1234 913 -972 -4536 908 -657 -2389 654 -576 -4114 202 -545 -8899 $76 $102 $54 $201 $58 $ nawk '/Ann/{print "tt. Have a nice day, " $1, $2 "!"}' employees Have a nice day, Steve Ann!

The printf Function n n The printf function can be used formatting fancy output The printf function returns a formatted string to standard output, like the printf statement in C. Unlike the print function, printf does not provide a newline. The escape, n, must be provided if a newline is desired When an argument is printed, the place where the output is printed is called the field, and when the width of the field is the number of characters contained in that field

The printf Function (continue. ) n Example 1: $ echo "UNIX" | nawk ' {printf "|%-15 s|n", $1}' |UNIX | $ echo "UNIX" | nawk '{printf "|%15 s|n", $1}' | $ UNIX|

The printf Function (continue. ) n Example 2: $ cat employees Chen Cho Tom Billy Larry White Bill Clinton Steve Ann 5/19/63 4/12/45 11/2/54 1/14/60 9/15/71 203 -344 -1234 913 -972 -4536 908 -657 -2389 654 -576 -4114 202 -545 -8899 $76 $102 $54 $201 $58 ID ID ID 5 4 11 1 9 $ nawk '{printf "The name is: %-15 s ID is %8 dn", $1, $3}' employees The The The $ name name is: is: is: Chen Tom Larry Bill Steve is is is

The printf Function (continue. ) Conversion Character Definition c Character s String d Decimal number ld Long decimal number u Unsigned decimal number lu Long unsigned decimal number

The printf Function (continue. ) Conversion Character Definition x Hexadecimal number lx Long hexadecimal number o Octal number lo Long octal number e Floating point number in scientific notation (e-notation) f Floating point number g Floating point number using either e or f conversion, whichever takes the least space

The printf Function (continue. ) Modifier Character Definition - Left-justification modifier # Integers in octal form are displayed with a leading 0; integers in hexadecimal form are displayed with a leading 0 x. + For conversions using d, e, f, and g, integers are displayed with a numeric sign + or - 0 The displayed value is padded with zeros instead of white space

The printf Function (continue. ) Printf Format Specifier What it Does Given x=‘A’, y=15, z=2. 3, and $1 = Bob Smith: %c Prints a single ASCII character. printf("The character is %cn", x) prints: The character is A %d %e %f Prints a decimal number printf("The boy is %d years oldn", y) prints: The boy is 15 years old Prints the e notation of a number printf("z is %fn", z) prints: z is 2. 3 e+01 Prints a floating point number printf("z is %en", z) prints: z is 4. 600000

The printf Function (continue. ) Printf Format Specifier What it Does Given x=‘A’, y=15, z=2. 3, and $1 = Bob Smith: %o Prints the octal value of a number printf("y is %on", y) prints: y is 16 %s Prints a string of characters %x printf("The name of the culprit is %sn", $1) prints: The name of the culprit is Bob Smith Prints the hex value of a number printf("y is %xn", y) prints: y is F

awk commands from within a file (continue. ) n n n If awk commands are placed in a file, the -f option is used with the name of the awk file, followed by the name of the input file to be processed A record is read into awk's buffer and each of the commands in the awk file are tested and executed for that record If an action is not controlled by a pattern, the default behavior is to print the entire record

awk commands from within a file (continue. ) n If a pattern does not have an action associated with it, the default is to print the record where the pattern matches an input line

awk commands from within a file (continue. ) n Example: Chen Tom Larry Bill Steve Cho Billy White Clinton Ann 5/19/63 4/12/45 11/2/54 1/14/60 9/15/71 203 -344 -1234 913 -972 -4536 908 -657 -2389 654 -576 -4114 202 -545 -8899 $ cat awkfile /^Steve/{print "Hello Steve!"} {print $1, $2, $3} $ nawk -f awkfile employees Chen Cho 5/19/63 Tom Billy 4/12/45 Larry White 11/2/54 Bill Clinton 1/14/60 Hello Steve! Steve Ann 9/15/71 $76 $102 $54 $201 $58

Records n By default, each line is called a record and is terminated with a newline

The Record Separator n n By default, the output and input record separator (line separator) is a carriage return, stored in the built-in awk variables ORS and RS, respectively The ORS and RS values can be changed, but only in a limited fashion

The $0 Variable n n n An entire record is referenced as $0 by awk When $0 is changed by substitution or assignment, the value of NF, the number of fields, may be changed The newline value is stored in awk's built-in variable RS, a carriage return by default

The $0 Variable (continue. ) n Example: $ cat employees Chen Cho Tom Billy Larry White Bill Clinton Steve Ann 5/19/63 4/12/45 11/2/54 1/14/60 9/15/71 203 -344 -1234 913 -972 -4536 908 -657 -2389 654 -576 -4114 202 -545 -8899 $76 $102 $54 $201 $58 $ nawk '{print $0}' employees Chen Cho Tom Billy Larry White Bill Clinton Steve Ann 5/19/63 4/12/45 11/2/54 1/14/60 9/15/71 203 -344 -1234 913 -972 -4536 908 -657 -2389 654 -576 -4114 202 -545 -8899 $76 $102 $54 $201 $58

The NR Variable n n The number of each record is stored in awk's built -in variable, NR After a record has been processed, the value of NR is incremented by one

The NR Variable (continue. ) n Example: $ cat employees Chen Cho Tom Billy Larry White Bill Clinton Steve Ann 5/19/63 4/12/45 11/2/54 1/14/60 9/15/71 203 -344 -1234 913 -972 -4536 908 -657 -2389 654 -576 -4114 202 -545 -8899 $76 $102 $54 $201 $58 $ nawk '{print NR, $0}' employees 1 2 3 4 5 Chen Cho Tom Billy Larry White Bill Clinton Steve Ann 5/19/63 4/12/45 11/2/54 1/14/60 9/15/71 203 -344 -1234 913 -972 -4536 908 -657 -2389 654 -576 -4114 202 -545 -8899 $76 $102 $54 $201 $58

Fields n n Each record consists of words called fields which, by default, are separated by white space, that is, blank spaces or tabs. Each of these words is called a field, an awk keeps track of the number of fields in its built-in variable, NF The value of NF can vary from line to line, and the limit is implementation-dependent, typically 100 fields per line

Fields (continue. ) n Example 1: $1 Chen Tom Larry Bill Steve $2 Cho Billy White Clinton Ann $3 5/19/63 4/12/45 11/2/54 1/14/60 9/15/71 $4 203 -344 -1234 913 -972 -4536 908 -657 -2389 654 -576 -4114 202 -545 -8899 $ nawk '{print NR, $1, $2, $5}' employees 1 2 3 4 5 Chen Cho $76 Tom Billy $102 Larry White $54 Bill Clinton $201 Steve Ann $58 $5 $76 $102 $54 $201 $58

Fields (continue. ) n Example 2: nawk '{print $0, NF}' employees Chen Cho 5/19/63 203 -344 -1234 Tom Billy 4/12/45 913 -972 -4536 Larry White 11/2/54 908 -657 -2389 Bill Clinton 1/14/60 654 -576 -4114 Steve Ann 9/15/71 202 -545 -8899 $76 $102 $54 $201 $58 5 5 5

The Input Field Separator • awk's built-in variable, FS, holds the value of the input field separator. • When the default value of FS is used, awk separates fields by spaces and/or tabs, stripping leading blanks and tabs • The FS can be changed by assigning new value to it, either: – in a BEGIN statement, or – at the command line

The Input Field Separator (continue. ) • To change the value of FS at the command line, the –F option is used, followed by the character representing the new separator

The Input Field Separator (continue. ) • Example: $ cat employees Chen Cho: 5/19/63: 203 -344 -1234: $76 Tom Billy: 4/12/45: 913 -972 -4536: $102 Larry White: 11/2/54: 908 -657 -2389: $54 Bill Clinton: 1/14/60: 654 -576 -4114: $201 Steve Ann: 9/15/71: 202 -545 -8899: $58 $ nawk -F: '/Tom Billy/{print $1, $2}' employees Tom Billy 4/12/45

Using More than One Field Separator • You may specify more than one input separator • If more than one character is used for the field separator, FS, then the string is a regular expression and is enclosed in square brackets • Example $ nawk -F'[ : t]' '{print $1, $2, $3}‘ employees Chen Cho 5/19/63 Tom Billy 4/12/45 Larry White 11/2/54 Bill Clinton 1/14/60 Steve Ann 9/15/71

The Output Field Separator • The default output field separator is a single space and is stored in awk's internal variable, OFS • The OFS will not be evaluated unless the comma separates the fields • Example: $ cat employees Chen Cho: 5/19/63: 203 -344 -1234: $76 Tom Billy: 4/12/45: 913 -972 -4536: $102 Larry White: 11/2/54: 908 -657 -2389: $54 Bill Clinton: 1/14/60: 654 -576 -4114: $201 Steve Ann: 9/15/71: 202 -545 -8899: $58 $ nawk -F: '/Tom Billy/{print $1 $2 $3 $4}' employees Tom Billy 4/12/45913 -972 -4536$102

Patterns • A pattern consists of – a regular expression, – an expression resulting in a true or false condition, or – a combination of these • When reading a pattern expression, there is an implied if statement

Patterns (continue. ) • Example: $ cat employees Chen Cho Tom Billy Larry White Bill Clinton Steve Ann 5/19/63 4/12/45 11/2/54 1/14/60 9/15/71 203 -344 -1234 913 -972 -4536 908 -657 -2389 654 -576 -4114 202 -545 -8899 $76 $102 $54 $201 $58 $ nawk '/Tom/' employees Tom Billy $ 4/12/45 913 -972 -4536 $102 nawk '$4 < 40' employees Chen Cho Steve Ann 5/19/63 203 -344 -1234 9/15/71 202 -545 -8899 $76 $58

Actions • Actions are statements enclosed within curly braces and separated by semicolons • Actions can be simple statements or complex groups of statements • Statements are separated – by semicolons, or – by a newline if placed on their own line

Regular Expressions • A regular expression to awk is a pattern that consists of characters enclosed in forward slashes • Example 1: $ nawk '/Steve/' employees Steve Ann 9/15/71 202 -545 -8899 $58 • Example 2: $ nawk '/Steve/{print $1, $2}' employees Steve Ann

Regular Expression Meta characters ^ Matches at the beginning of string $ Matches at the end of string . Matches for a single character * Matches zero or more of preceding character + Matches for one or more of preceding character ? Matches for zero or one of preceding character [ABC] Matches for any one character in the set of characters, i. e. , A, B, or C

Regular Expression Meta characters (continue. ) [^ABC] Matches characters not in the set of characters, i. e. , A, B or C [A-Z] Matches for any character in the range from A to Z A|B Matches either A or B (AB)+ Matches one or more sets of AB * Matches for a literal asterisk & Used in the replacement to represent what was found in the search string

Regular Expressions (continue. ) • Example 3: $ nawk '/^Steve/' employees Steve Ann 9/15/71 202 -545 -8899 $58 • Example 4: $ nawk '/^[A-Z][a-z]+ /' employees Chen Cho Tom Billy Larry White Bill Clinton Steve Ann 5/19/63 203 -344 -1234 4/12/45 913 -972 -4536 11/2/54 908 -657 -2389 1/14/60 654 -576 -4114 9/15/71 202 -545 -8899 $76 $102 $54 $201 $58

The Match Operator • The match operator, the tilde (~), is used to match an expression within a record or a field • Example 1: $ cat employees Chen Cho Tom Billy Larry White Bill Clinton Steve Ann 5/19/63 4/12/45 11/2/54 1/14/60 9/15/71 203 -344 -1234 913 -972 -4536 908 -657 -2389 654 -576 -4114 202 -545 -8899 $76 $102 $54 $201 $58 $ nawk '$1 ~ /[Bb]ill/' employees Bill Clinton 1/14/60 654 -576 -4114 $201

The Match Operator (continue. ) • Example 2: $ nawk '$1 !~ /lee$/' employees Chen Cho Tom Billy Larry White Bill Clinton Steve Ann 5/19/63 4/12/45 11/2/54 1/14/60 9/15/71 203 -344 -1234 913 -972 -4536 908 -657 -2389 654 -576 -4114 202 -545 -8899 $76 $102 $54 $201 $58

awk Commands in a Script File • When you have multiple awk pattern/action statements, it is often easier to put the statements in a script • The script file is a file containing awk comments and statements • If statements and actions are on the same line, they are separated by semicolons • Comments are preceded by a pound (#) sign

awk Commands in a Script File (continue. ) • Example: $ cat employees Chen Cho: 5/19/63: 203 -344 -1234: $76 Tom Billy: 4/12/45: 913 -972 -4536: $102 Larry White: 11/2/54: 908 -657 -2389: $54 Bill Clinton: 1/14/60: 654 -576 -4114: $201 Steve Ann: 9/15/71: 202 -545 -8899: $58 $ cat info # My first awk script by Abdelshakour Abuzneid # Script name: info; Date: February 09, 2001 /Tom/{print "Tom's birthday is "$3} /Bill/{print NR, $0} /^Steve/{print "Hi Steve. " $1 " has a salary of " $4 ". "} #End of info script

awk Commands in a Script File (continue. ) • Example (continue. ): $ nawk -F: -f info employees Tom's birthday is 913 -972 -4536 2 Tom Billy: 4/12/45: 913 -972 -4536: $102 4 Bill Clinton: 1/14/60: 654 -576 -4114: $201 Hi Steve Ann has a salary of $58. • To view info script, click here

Comparison Expressions • Comparison expressions match lines where if the condition is true, the action is performed • The value of the expression evaluates true, and 0 if false

Relational Operators Operator Meaning Example < <= == != >= > ~ !~ Less than or equal to Equal to x<y x <= y x == y Not equal to Greater than or equal to Greater than Matched by regular expression Not matched by regular expression x x x != y >y ~ / y/ !~ /y/

Relational Operators (continue. ) • Example: $ cat employees Chen Cho Tom Billy Larry White Bill Clinton Steve Ann 5/19/63 4/12/45 11/2/54 1/14/60 9/15/71 203 -344 -1234 913 -972 -4536 908 -657 -2389 654 -576 -4114 202 -545 -8899 76 102 54 201 58 Bill Clinton 1/14/60 654 -576 -4114 201 Tom Billy 4/12/45 913 -972 -4536 102 Steve Ann 9/15/71 202 -545 -8899 58 $ nawk '$5 == 201' employees $ nawk '$5 > 100' employees $ nawk '$2 ~ /Ann/ ' employees

Relational Operators (continue. ) • Example (continue): $ nawk '$2 !~ /Ann/ ' employees Chen Cho Tom Billy Larry White Bill Clinton 5/19/63 4/12/45 11/2/54 1/14/60 203 -344 -1234 913 -972 -4536 908 -657 -2389 654 -576 -4114 76 102 54 201

Conditional Expressions • A conditional expression uses two symbols, the question mark and the colon, to evaluate expression • Format: conditional expression 1 ? expression 2 : expression 3

Conditional Expressions (continue. ) • Example: $ nawk '{max=($1 > $2) ? $1 : $2; print max}' employees Cho Tom White Clinton Steve

Computation • awk performs all arithmetic in floating point Operator + * / % ^ Meaning Add Substract Multiply Divide Modulus Exponentiation Example x+y x–y x*y x/y x%y x^y

Computation (continue. ) • Example: $ nawk '$3 * $4 > 500' filename

Compound Patterns • Compounds patterns are expressions that combine patterns with logical operators Operator && || ! Meaning Logical AND Logical OR NOT Example a && b a || b !a

Compound Patterns (continue. ) • Example : $ nawk '$2 > 5 && $2 <= 15' employees $ $ nawk '$5 == 1000 || $3 > 50' employees Steve Ann $ 9/15/71 202 -545 -8899 58

Range Patterns • Range patterns match from the first occurrence of one pattern to the first occurrence of the second pattern, then match for the next occurrence of the second pattern, etc • If the first pattern is matched and the second pattern is not found, awk will display all lines to the end of the file • Example : $ nawk '/Tom/, /Steve/' employees Tom Billy Larry White Bill Clinton Steve Ann $ 4/12/45 11/2/54 1/14/60 9/15/71 913 -972 -4536 908 -657 -2389 654 -576 -4114 202 -545 -8899 102 54 201 58

Numeric and String Constants • Numeric constants can be represented as – Integer like 243 – Floating point numbers like 3. 14, or – Numbers using scientific notation like. 723 E-1 or 3. 4 • Strings, such as Hello are enclosed in double quotes

Initialization and Type Coercion • A variable can be – a string – a number, or – both • When it is set, it becomes the type of the expression on the right-hand side of the equal sign • Initialized variables have the value zero or the value " ", depending on the context in which they are used

User-Defined Variables • User-defined variables consist of letters, digits, and underscores, and cannot begin with a digit • Variables in awk are not declared • If the variable is not initialized, awk initializes string variables to null and numeric variables to zero • Variables are assigned values with awk's assignment operators • Example : $ nawk '$1 ~ /Tom/ {wage = $5 * 40; print wage}' employees 4080

Increment and Decrement Operators • The expression x++ is equivalent to x=x+1 • The expression x– is equivalent to x=x-1 • You can use the increment and decrement operators either preceding operator, as in ++x, or after the operator, as x++ {x = 1; y = x++; print x, y} name=“Nancy” name is string x++ x is a number; x is initialized to zero and incremented by 1 number=35 number is a number

User-Defined Variables at the Command line • A variable can be assigned a value at the command line and passed into awk script • Example: $ nawk –F: -f awkscript month=4 filename year=1999

The –v Option (nawk) • The –v option provided by nawk allows command line arguments to be processed within a BEGIN statement • For each argument passed at the command line, there must be a –v option preceding it

Built-in Variables • Built-in variables have uppercase names. They can be used in expressions and can be reset Variable Name Variable Contents ARGC Number of command line argument ARGV Array of command line arguments FILENAME Name of current input file FNR Record number in current file FS The input field separator, by default a space

Built-in Variables (continue. ) Variable Name Variable Contents NF NR OFMT Number of fields in current record OFS Output field separator ORS RLENGTH RS Output record separator RSTART Offset of string matched by match function SUBSEP Subscript separator Number of record so far Output format for numbers Length of string matched by match function Input record separator

Built-in Variables (continue. ) • Example: $ nawk -F: '$1 == "Steve Ann"{print NR, $1, $2, $NF}' employees 2 5 Steve Ann 9/15/71 $58

BEGIN Patterns • The BEGIN pattern is followed by an action block that is executed before awk processes any lines from the input file • The BEGIN action is often used to change the value of the built-in variables, OFS, RS, FS, and so forth, to assign initial values to user-defined variables, and to print headers or titles as part of the output

BEGIN Patterns (continue. ) • Example 1 : $ nawk 'BEGIN{FS=": "; OFS="t"; ORS="nn"}{print $1, $2, $3}' employees 2 Chen Cho 5/19/63 203 -344 -1234 Tom Billy 4/12/45 913 -972 -4536 Larry White 11/2/54 908 -657 -2389 Bill Clinton 1/14/60 654 -576 -4114 Steve Ann 9/15/71 202 -545 -8899 $

BEGIN Patterns (continue. ) • Example 2 : $ nawk 'BEGIN{print "Make Year"}'

END Patterns • END patterns do not match any input lines, but executes any actions that are associated with the END patterns are handled after all lines of input have been processed • Examples: $ nawk 'END{print "The number of records is " NR }' employees The number of records is 5 $ nawk '/Steve/{count++}END{print "Steve was found " count " times. "}' employees Steve was found 1 times. $

Output Redirection • When redirecting output from within awk to a UNIX file, the shell redirection operators are used • The filename must be enclosed in double quotes • Once the file is opened, it remains opened until explicitly closed or the awk program terminates • Example: $ nawk '$5 >= 70 {print $1, $2 > "passing_file" }' employees $ cat passing_file Chen Cho Tom Billy Bill Clinton

The getline Function • Reads input from – The standard input, – a pipe, or – a file other than from the current file being processed • It gets the next line of input and sets the NF, NR and the FNR built-in variables • The getline function returns – 1 if a record is found – 0 if EOF (end of file) – -1 if there is an error

The getline Function (continue. ) • Examples : $ nawk 'BEGIN{ "date" | getline d; print d}' employees 2 Fri Feb 9 09: 39: 53 EST 2001 $ nawk 'BEGIN{ "date" | getline d; split( d, mon); print mon[2]}' employees Feb $ nawk 'BEGIN{while("ls" | getline) print} UNIX varfile 2 varfile 3 varfile 4 varfile 5 varfile 6

The getline Function (continue. ) • Examples : $ nawk 'BEGIN{ print "What is your name? " ; > getline name < "/dev/tty"} > $1 ~ name {print "Found " name " on line ", NR ". "} > END{print "See ya, " name ". "}' employees What is your name? abdul See ya, abdul. $

Pipes • If you open a pipe in an awk program, you must close it before opening another one • The command on the right-hand side of the pipe symbol is enclosed in double quotes

Pipes (continue. ) • Example: $ cat names jhon smith alice cheba tony tram dan savage eliza goldborg $ nawk '{print $1, $2 | "sort -r +1 -2 +0 -1 "}' names tony tram jhon smith dan savage eliza goldborg alice cheba $

Closing Files and Pipes • The pipe remains opened until awk exits • Statements in the END block will also be affected by the pipe. The first line in the END block closes the pipe • Example: ( In script) { print $1, $2, $3 | “ sort –r +1 – 2 +0 – 1”} END{ Close(“sort –r +1 – 2 +0 – 1”) <rest of statement> }

The System Function • The built-in system function takes a UNIX (operating system command) command as its argument, executes the command, and returns the exit status to the awk program • The UNIX command must be enclosed in double quotes • Example: ( In script) { System ( “cat “ $1 ) System ( “clear” ) }

If Statement • Format: If (expression) { statement; … }

If/else Statement • Format: {If (expression) { statement; … } else { statement; … } }

If/else Statement • Example: $ nawk '{if($6 > 50) print $1 "Too high"; > else print "Range is OK"}' names Range Range $ is is is OK OK OK

If/else if Statement • Format: { If (expression) { statement; … } else if (expression) { statement; … } else { statement; … } }

Loops • Loops are used to iterate through the field within a record and to loop through the elements of an array in the END block

While Loop • The first step in using a while loop is to set a variable to an initial value • The do/while loop is similar to the while loop, except that the expression is not tested until the body of the loop is executed at least once

While Loop (continue. ) • Example: $ nawk '{ i = 1; while (i <= NF ) { print NF, $i; i++}}' names 2 2 2 2 2 $ jhon smith alice cheba tony tram dan savage eliza goldborg

for Loop • for loop requires three expressions within the parentheses: the initialization expression, the test expression and the expression to update the variables within the test expression • The first statement within the parentheses of the for loop can perform only one initialization

for Loop (continue. ) • Example: $ nawk '{ i = 1; while (i <= NF ) { print NF, $i; i++}}' names 2 2 2 2 2 $ jhon smith alice cheba tony tram dan savage eliza goldborg

break and continue Statement • The break statement lets you break out of a loop if a certain condition is true • The continue statement causes the loop to skip any statement that follow if a certain condition is true, and returns control to the top of the loop, starting at the next iteration • Example: (In Script) { if ($1 Peter} {next}} else {print} }

next Statement • The next statement gets the next line of input from the input file, restarting execution at the top of the awk script • Example: (In Script) 1 {for ( x = 3; x <= NF; x++) if ( $x < 0) {print “Bottomed out!”; break} # breaks out of the loop } 2 {for ( x = 3; x <= NF; x++ ) if ( $x == 0 ) { print “Get next item”; continue} # starts next iteration of the for loop }

exit Statement • The exit statement is used to terminate the awk program. It stops processing records, but does not skip over an END statement • If the exit statement is given a value between 0 and 255 as an argument (exit 1), this value can be printed at the command line to indicate success or failure by typing:

exit Statement (continue. ) • Example: (In Script) {exit (1)} (The Command Line) % echo $status (csh) 1 $ echo $? (sh/ksh) 1

Arrays • Arrays in awk are called associative arrays because the subscripts can be either – number, or – string • The keys and values are stored internally is a table where a hashing algorithm is applied to the value of the key in question • An array is created by using it, and awk can infer whether or not is used to store numbers or strings

Arrays (continue. ) • Array elements are initialized with – numeric value, and – string value null • You do not have to declare the size of an aw array • awk arrays are used to collect information from records and may be used for accumulating totals, counting words, tracking the number of times a pattern occurred

Arrays (continue. ) • Example: $ cat employees Chen Cho Tom Billy Larry White Bill Clinton Steve Ann 5/19/63 4/12/45 11/2/54 1/14/60 9/15/71 203 -344 -1234 913 -972 -4536 908 -657 -2389 654 -576 -4114 202 -545 -8899 76 102 54 201 58 $ nawk '{name[x++]=$2}; END{for(i=0; i<NR; i++) > print i, name[i]}' employees 0 1 2 3 4 Cho Billy White Clinton Ann

Arrays (continue. ) • Example: $ nawk '{id[NR]=$3}; END{for(x = 1; x<= NR; x++) > print id[x]}' employees 5/19/63 4/12/45 11/2/54 1/14/60 9/15/71 $

The special for Loop • The special for loop is used to read through an associative array when strings are used as subscripts or the subscripts are not consecutive numbers • When strings are used as subscripts or the subscripts are not consecutive numbers

The special for Loop (continue. ) • Example: $ cat db Tom Jones Mary Adams Sally Chang Billy Black Tom Savage $ nawk '/^Tom/{name[NR]=$1}; > END{for( i=1; i <= NR; i++ )print name[i]}' db Tom $

The special for Loop (continue. ) • Example: $ nawk '/^Tom/{name[NR]=$1}; > END{for(i in name){print name[i]}}' db Tom $

Using Strings as Array Subscripts • A subscript may consist of a variable containing a string or literal string • If the string is a literal, it must be enclosed in double quotes • Example $ cat db Tom Jones Mary Adams Sally Chang Billy Black Tom Savage $ nawk -f awkscript db There are 2 Tom's in the file and 1 Mary's in the file.

Using Field Values as Array Subscripts (continue. ) • Any expression can be used as a subscript in an array. Therefore, fields can be used • Example 1: $ cat db Tom Jones Mary Adams Sally Chang Billy Black Tom Savage $ nawk '{count[$2]++}END{for(name in count)print name, count[name] }' db Chang 1 Black 1 Jones 1 Savage 1 Adams 1 $

Using Field Values as Array Subscripts (continue. ) • Example 2: $ cat db Tom Jones Mary Adams Sally Chang Billy Black Tom Savage $ nawk '{count[$2]++}END{for(name in count)print name, count[name] }' db Chang 1 Black 1 Jones 1 Savage 1 Adams 1 $

Arrays and the split Function • awk’s built-in split function allows you to split string into words and store them in an array • You can define the field separator or use the value currently stored in FS • Format: split(string, array, field separator) split(string, array)

The delete Function • The delete function removes an array elements

Multidimensional Arrays: Nawk • Multidimensional array is done by concatenating the indices into a string separated by the value of a special built-in variable, SUBSEP • The SUBSEP variable contains the value “34”, an unprintable character • The expression matrix[2, 8] is really the array matrix[2 SUBSEP 8] which evaluates to matrix[“ 2348”]

ARGV • Command line arguments are available to nawk with the built-in array called ARGV • These arguments include the command nawk, but not any of the options passed to nawk • The index of the ARGV array starts at zero

ARGC • ARGC is a built-in variable that contains the number of command line arguments • Example: $ cat myscript #This script is called myscript BEGIN{ for ( i = 0; i< ARGC; i++){ printf("argv[%d] is %sn", i, ARGV[i]) } printf("The number of arguments, ARGC=%dn", ARGC) }

ARGC (continue. ) • Example : $ cat ARGVS #This script is called argvs BEGIN{ for ( i=0; i < ARGC; i++ ) { printf("argv[%d] is %sn", i, ARGV[i]) } printf("The number of arguments, ARGC=%dn", ARGC) } $ $ nawk -f ARGVS datafile argv[0] is nawk argv[1] is datafile The number of arguments, ARGC=2 $

ARGC (continue. ) • Example : $ nawk -f ARGVS datafile "Peter Pan" 12 argv[0] is argv[1] is argv[2] is argv[3] is The number $ nawk datafile Peter Pan 12 of arguments, ARGC=4

ARGC (continue. ) • Example : $ cat arging #This script is called argin BEGIN{FS=": "; name=ARGV[2] print "ARGV[2] is "ARGV[2] } $1 ~ name { print $0 } $ nawk -f arging employees 2 "Chen Cho" ARGV[2] is Chen Cho: 5/19/63: 203 -344 -1234: $76 nawk: can't open file Chen Cho input record number 5, file Chen Cho source line number 1

ARGC (continue. ) • Example : $ cat arging 1 #This script is called argin BEGIN{FS=": "; name=ARGV[2] print "ARGV[2] is "ARGV[2] delete ARGV[2] } $1 ~ name { print $0 } $ nawk -f arging 1 employees 2 "Chen Cho" ARGV[2] is Chen Cho: 5/19/63: 203 -344 -1234: $76

The sub and gsub Functions • The sub function matches the regular expression for the largest and leftmost substring in the record, and replaces that substring with the substitution string • If a target string is specified, the regular expression is matched for the largest and leftmost substring in the target string, and the substring is replaced with the substitution string • If a target string is not specified, the entire record is used

The sub and gsub Functions (continue. ) • Format: sub (regular expression, substitution string); sub (regular string, substitution string, target string) • Format: gsub (regular expression, substitution string) sub (regular string, substitution string, target string)

The index Function • The index function returns the first position where a substring is found in a string • Offset starts at position 1

The length Function • The length function returns the number of characters in a string • Without an argument, the length function returns the number of characters in a record • Format: length (string) length

The length Function • Example $ nawk '{ print length("hello") }' employees 5 5 5 $

The substr Function • The substr function returns the substring of a string starting at a position where the first position is one • If the length of the substring is given, that part of the string is returned • If the specified length exceeds the actual string, the string is returned • Format: substr (string, starting position) substr (string, starting position, length of string

The match Function • The match function returns the index where the regular expression is found in the string, or zero if not found • The match function sets the built-in variable RSTART to the starting position of the substring within the string, and RLENGTH to the number of characters to the end of the substring • Format: match (string, regular expression)

The match Function (continue. ) • Example : $ nawk 'END{start=match("Good ole USA", /[AZ]+$/); > print RSTART, RLENGTH}' employees 10 3

The split Function • The split function splits a string into an array using whatever field separator is designated as the third parameter • If the third parameter is not provided, awk will use the current value of FS • Format: split (string, array, field separator) split (string, array)

The sprintf Function • The sprintf function returns an expression in a specified format. It allows you to apply the format specifications of the printf function • Format: Variable=sprintf(“string with format specifiers” expr 1, expr 2, …, exprn)

The sprintf Function (continue. ) • Example : $ awk '{line = sprintf ( "%-15 s %6. 2 f ", $1 , $3 ); > print line}' employees Chen Tom Larry Bill Steve 0. 00

Built-in Arithmetic Functions Name Value returned atan 2(x, y) Arctangent of y/x in the range cos(x) Cosine of x, with x in radians exp(x) Exponential function of x, e int(x) Integer part of x; truncated toward 0 when x>0 log(x) Natural (base e) logarithm of x rand() Random number r, where 0<r<1

Built-in Arithmetic Functions (continue. ) Name sin() sqrt(x) srand(x) Value returned Sine of x, with x radians Square root of x X is a new seed for rand()

Integer Function • The int function truncates any digits to the right of the decimal point to create a whole number. There is no rounding off

The rand Function • The rand function generates a pseudorandom floating point number greater than or equal to zero and less than one • Example $ nawk '{print rand()}' employees 0. 513871 0. 175726 0. 308634 0. 534532 0. 94763

The srand Function • The srand function without an argument uses the time of day to generate the seed for the rand function • srand(x) uses x as the seed. Normally, x should vary during the run of the program • Example $ nawk 'BEGIN{srand()}; {print rand()}' employees 0. 548753 0. 392254 0. 972472 0. 821497 0. 153722 $

The srand Function (continue. ) • Example $ nawk 'BEGIN{srand()}; {print rand()}' employees 0. 548753 0. 392254 0. 972472 0. 821497 0. 153722 $ nawk 'BEGIN{srand()}; {print rand()}' employees 0. 00537126 0. 312784 0. 23722 0. 132023 0. 17304

User-Defined Functions (nawk) • A user-defined function can be placed anywhere in the script that a pattern action rule can • Format function name (parameter, …){ statements return expression (the return statement and expression are optional) }

User-Defined Functions (nawk) (continue. ) • Variables are passed by value and are local to the function where they are used • Arrays are passed by address or by references, so array elements can be directly changed within the function • Any variable used within the function that has not been passed in the parameter list is considered a global variable that is, it is visible to the entire awk program, and if changed in the function, is changed throughout the program

User-Defined Functions (nawk) (continue. ) • • The only way to provide local variables within a function is to include them in the parameter list If there is not a formal parameter provided in the function call, the parameter is initially set to null The return statement returns control and possibly a value to the caller Example $ cat grades 44 55 66 22 77 99 100 22 77 99 33 66 55 66 100 99 88 45 $

User-Defined Functions (nawk) (continue. ) • Example $ cat sorter #Script is called sorter # It sorts numbers in ascending order function sort ( scores, num_elements, temp, i, j ) { # temp, i, and j will be local and private, #with an initila value of null. for( i = 2; i <= num_elements ; i++ ) { for ( j = i; scores [j-1] > scores[j]; --j) { temp = scores[j] = scores[j-1] = temp } } }

User-Defined Functions (nawk) (continue. ) • Example {for ( i = 1; i <= NF; i++) grades[i]=$i sort(grades, NF) for (j = 1; j <= NF; j++ ) printf( "%d ", grades[j] ) printf("n") } $ nawk -f sorter grades 22 44 55 66 77 99 22 33 66 77 99 100 45 55 66 88 99 100 • To view sorter script, click here

The substr function • In the following example, the fields are of fixed width, but are not separated by a field separator. The substr function is used to create field

Empty Fields • If the data is stored in fixed-width fields, it is possible that some of the fields are empty. In the following example, the substr function is used to preserve the fields, whether or not they contain data • Example $ cat file xxx xxx abc xxx a bbb xxx xx

Empty Fields (continue. ) $ cat awkfix #Preserving empty fields. Field width is fixed. { f[1]=substr($0, 1, 3) f[2]=substr($0, 5, 3) f[3]=substr($0, 9, 3) line=sprintf("%-4 s%-4 sn", f[1], f[2], f[3]) print line } $ nawk -f awkfix file xxx xxx abc xxx a bbb xxx xx

Empty Fields (continue. ) • To view awkfix script, click here

References n n n UNIX SHELLS BY EXAMPLE BY ELLIE QUIGLEY UNIX FOR PROGRAMMERS AND USERS BY G. GLASS AND K ABLES UNIX SHELL PROGRAMMING BY S. KOCHAN AND P. WOOD