UNIX Toolbox Philosophy 1 Create powerful command line

UNIX Toolbox Philosophy 1. Create powerful command line tools – – Each does one thing Each does it very well 2. Design these tools to work well together 3. The tools handle streams using pipes – – – Filters are commands whose inputs and outputs are stream Job is to transform the input is either from stdin or a file 4. The tools are normally C programs

Bash Command Processing 1. Read a command 2. Transform the command • • • Process history substitutions Process alias substitutions Process special character (break sequences) Process variable substitutions Process command substitutions Process pathname expansions 3. Execute the Command 4. Back to step 1 to get next command Note: transformations occur in the order stated above

History Command • UNIX records all of the commands executed • Use the up arrow to step through previously entered commands; the down arrow steps forward • $HISTFILE contains the name of the hidden file with a log of previously entered commands • Ctrl R <keyword> finds the previous command entered with <keyword> • The history command – displays all previous commands starting with its # identifer. – !# will execute the command with the # identifier – history > commands creates a file with a record of past executed commands

Bash History • • • Turn on History command: export HISTSIZE=20 Clear history list: history -c History list: >history Execute nth history command: >history >!3 Execute last command: >grep Bush grades >!! | awk '{print $4}' Use arguments from the last command: echo !* Use last argument of last command: >cp prog. c new. c >ls !$ Use the first argument of last command: >ls –l >echo !^ Navigate around history commands: use arrow keys Note: argument is anything following the command name

The script command • The script command is handy >script OR >script. File • It logs all your input and system output to a file • If you do not specify a file, the log goes to a file called typescript • Type >exit to terminate the logging

Pipes • Definition: The facility to direct the output of one command to the input of the next (cmd 1 | cmd 2) • The pipe character: | • Example: ls -R ~ | grep a* | sort | uniq List all unique file names starting with a in alphabetic order • Example: ls –al | less Directory listing enabling us to page through the result

Streams and Redirection Stream: a flow of data from source to sink • • • C programs have default stream sources or sinks – 0 or stdin: standard input (keyboard) – 1 or stdout: standard output (monitor) – 2 or stderr: error output (often same as stdout) Redirection: alter the default source or sink Examples – – – – – Redirect stdout to a file: ls ~ > file. Name Redirect a file list and append to a file: ls ~ >> filename Redirect input from a file instead of stdin: sort < file_list. txt Redirect error messages: grep harley ~ 2> file Redirect both stdout and stderr: grep harley ~ &> file Redirect stdout, stderr: ls –r / 1>out 2>err Redirect stdout, stderr – ignore errors: ls –r / 1>out 2>dev/null Redirect, append stdout and stderr: grep harley ~ &>>file Create an empty file: >file Type into an empty file: cat

Shell Command Parsing • The shell transforms commands before they are executed • Examples – >echo $TERM replaces the variable with its contents – >ls x* replaces the * with a list of files (in this case starting with x)

Pathname Expansions • • Any string of 0 or more length: * Any single character: ? Any single character specified: [abc] Examples: – ls cs 367/*. c – ls cs 367*c – ls newfile? ? ? – ls out[12] Notes: Filenames must not have an initial period or slashes

ls Examples using Wildcards • * means any character sequence – ls cs 367/*. c, ls 367*c, ls new. File* • ? Means any single character – ls new. File? ? ? • [characters] means any of the characters – ls newfile[12] • [char-char] means any within range – ls newfile[0 -9][0 -9] • [: class: ] means files matchin predefined class – ls [: alpha: ], ls [: digit: ], [: lower: ]

More Command Line Expansions • Arithmetic expansion: use command line as a calculator (cumbersome syntax) echo $( (2+3) ) echo $( (5**3) ) % 2) ) • Brace expansion: simple loop of names echo Front-{A, B, C}-back Output: Front-A-back Front-B-back Front-C-back echo a. . c Output: a b c • Command substitution: command output becomes an argument to another command echo $(pwd) or echo`pwd` Output: /home/SOU/harveyd/cs 367

File Permissions • Ls –l format: -rw-r--r-- 1 owner group 213 Aug 26 16: 31 README • The line contains 1. 2. 3. 4. 5. 6. • • - for a regular file or d for a directory permission string (user, group, and other (world)) Number of links to file or directory the file size in decimal right-justified in a 13 -byte field; modification data file name Permission groups: u=user, g=group, o=other (world), a=all Permission types: read [r (4)], write/delete [w (2)], execute [x (1)]

Change File Mode (chmod) Purpose: Change file permissions • Examples: – Write for user and group, read for world >chmod ug+w, o+r trak – All permissions for user, read and execute for group and world: >chmod u+rwx, go+rx file. cgi • Remove execute permission to user, read, and world: >chmod a-x file. Name – An easier approach is to the use octal bits for (ogw) – For example, read, write, execute for owner, read, write for group and read only for world >chmod 765 file. Name

Concatenating files • Method one (cp command) >cp file 1 file 2 file 3 destination • Method two (cat command with redirection) cat jnk. txt grades >both. txt Note: cat is the command to type the contents of a file • Method three (concatenation redirection) – cat jnk. txt >both. txt – cat grades >>both. txt – cat >abc allows keyboard entry into a file called abc

Parts of files: head and tail • Display the first five lines of a file: head –n 5 • Display the last three lines of a file: tail –n 3 file • Copy first file lines of file to new. File: head –n 5 file >new. File

Inhibit Browser Command Line Expansion • Problem: we might want to override special characters and have UNIX interpret them literally • Example >echo $HOST ~ echoes the host computer plus the home directory name >echo $HOST ~ echoes the literal string >echo '$HOST ~' echoes everything in quotes literally >echo "$HOST ~" echoes everything except for and $

The tar command • • Problem: want to copy a bunch of files at once Solution: tar glues a series of files together Note: this is not compression tar commands – – Glue files together: >tar –cf tarfile. List Verbose output: tar -cvf tar. File file. List files in a tar. File: tar -tf tar. File Extract files from tar. File: >tar -xvf tar. File • Note: The dash is not needed in the above commands

The compress utilities • Compression – greatly reduces the size of the files being transferred – Caution: Compressing a compressed file can make it bigger • Commands – Compress a file and create a. gz extension (gzip <file>) – Uncompress a file (gunzip <file>. gz or gzip –d <file>) – Uncompressed tar archive (tar zxvf <file>. tar. gz) Or > tar zxvf file. Name. tar. gz • Syncing directories (if rsync is installed) – Make sure that second directory is “synced” with the first – rsync -av --delete <first> <second> Note: bzip 2/bunzip 2 work like gzip/gunzip, with another compression algorithm Note: zip/unzip are used to compress and un-compress as done in Windows

Managing Processes • Display processes and pid number >ps displays all of your processes >ps a removes "only yourself" restriction >ps u display in a user friendly format >ps au combines the above options >ps –l displays your processes in long format • Kill a process – Force without graceful termination (kill -9 <pid>) – Graceful termination (kill <pid> or killall <name>) • Monitor running processes (top) Note: the above ps options should not be preceded with a dash (-)

Printing and transferring • Printing >lpr filename >lpr –Pprinter filename >lpq (see what is in the print queue) >lpstat (status of printers) • The sftp program allows secure file transfers >sftp hostname Then enter commands on the remote system like cd, ls, put, get, quit

Foreground, Background, Suspend • Concepts – A single foreground process can run within the shell – Many background processes can run outside the shell – A suspended process stops running till it is restarted • Commands – – – – suspend process and move to background: <ctrl>z track background processes: >jobs resume last suspended process in background: >bg resume nth background process: >bg %n move last suspended process to foreground: >fg move nth background process to foreground: fg %n run a command in the background >sort file &

Simple Text Processing More Later • grep ‘regular expression’ file… • Simple examples (Search file for the string abc) – – Display all matching lines: grep abc file Display all, ignoring case: grep -i abc file Count lines that match: grep –c abc file Prefix with line numbers: grep –n abc file • Count lines, words, characters, bytes – – – Show count of words, characters, bytes: wc file Show only word count: wc -w file Show number of lines: wc -l file Show only byte count: wc -c file Show character count: wc -m file

Editors: History • ed: a line editor (Ken Thompson) • ex: An improved version of ed (Bill Joy) • sed: stream editor, can apply editing commands from the command line using streams • vi: Screen oriented text editor (We will use this one) – – Standard UNIX editor available on all UNIX systems smaller and faster than emacs minimal keystrokes to do tasks, hard to learn • emacs: (Richard Stallman) – powerful, cumbersome, part of the GNU IDE • Others: these work more like those on windows, but they are not guaranteed to be on every UNIX system

vi editor • Enter and exit the program – To enter: >vi file. Name – To exit: colon and then wq (write and quit) or q! (quit without write) • Modes – insert mode: input text into a file – command mode: keyboard command key – last line mode: longer commands input command key esc command : or / esc last-line

File Pointer • All commands are relative to a single file pointer • Central to using vi is navigating around the file • Navigation commands: – – – arrows work as you would expect on most vi Go to line 1: 1 G Go to line 50: 50 G Go forward one word: w Go to the end of the line: $ Go to the beginning of the line: ^

Delete, Cut, Copy, Paste • • • delete one word: dw delete 4 words: 4 dw or d 4 w delete (cut) 50 lines 50 dd yank (copy the current line: yy yank (copy) 50 lines: 50 yy yank (copy) into one of 26 buffers buffer c: "cy paste cut/copied text after the current line: P paste cut/copied text before the current line: p paste from buffer c: "cp delete one character: x delete 50 characters: 50 x

More vi Commands • • Undo command after a mistake: u Undo all recent commands: U join this line and the next: j joint four lines: 4 j replace a single character: r Save and exit: ZZ Change the tab stops: set tabstop=4 Entering unix commands: sh

vi Insert Mode Examples • • change rest of a line until <esc>: c$ Change a word until <esc>: cw Change line until <esc>: cc Insert before file pointer until you <esc>: i Append after file pointer until <esc>: a Insert below until <esc>: o Insert above until <esc>: O Replace until <esc>: R

vi Last-Line examples • • • Search for the word cat: /cat Search backwords for cat: ? cat Replace all this line: cat by dog: s/cat/dog/g Replace once: cat by dog: s/cat/dog/ Replace on every line: %s/cat/dog/ Replace all every line: %s/cat/dog/g Save the file: w Save the file as: w file. Name Save and quit: wq Merge a file into the current place: r file. Name Display line numbers: set number Don't display line numbers: set nonumber