Basic Text Processing Redirection and Pipes Lecture Overview

Basic Text Processing, Redirection and Pipes

Lecture Overview Basic text processing commands n head, tail, wc Redirection and pipes Getting to know vi Basic vi commands Advanced vi commands 2

Text Processing Commands UNIX is strongly text oriented It contains many utilities to search, modify or view text files This provides a simple mechanism for storing data and for passing it between applications and utilities 3

Viewing Portions of a File – head and tail The head command shows only the first few lines of a file head [options] [files] With no options, head shows first 10 lines Command line options for head: n n -N – specify number of lines to show (N is a number, not the letter N) -c. N – show first N bytes, not lines 4

Viewing Portions of a File – head and tail The tail command is similar to head, but shows the last lines of the file In addition, tail provides these options: n n +N – show lines starting with line N -f – output appended data as the file grows. With this option, the number of lines printed keeps growing as the file grows 5

head and tail – Examples Given a file called 'my_phones. txt': ADAMS, Andrew 7583 BARRETT, Bruce 6466 BAYES, Ryan 6585 BECK, Bill 6346 BENNETT, Peter 7456 GRAHAM, Linda 6141 HARMER, Peter 7484 MAKORTOFF, Peter 7328 MEASDAY, David 6494 NAKAMURA, Satoshi 6453 REEVE, Shirley 7391 ROSNER, David 6830 6

head and tail – Examples head -3 my_phones. txt ADAMS, Andrew 7583 BARRETT, Bruce 6466 BAYES, Ryan 6585 tail -3 my_phones. txt NAKAMURA, Satoshi 6453 REEVE, Shirley 7391 ROSNER, David 6830 head –c 30 my_phones. txt ADAMS, Andrew 7583 BARRETT, Br 7

head and tail – Examples tail +9 my_phones. txt MEASDAY, David 6494 NAKAMURA, Satoshi 6453 REEVE, Shirley 7391 ROSNER, David 6830 tail –c 50 my_phones. txt toshi 6453 REEVE, Shirley 7391 ROSNER, David 6830 8

Counting Words – wc The wc command counts the number of bytes, words and lines in a file wc [options] [files] Command line options for wc: n -c – print the number of bytes n -w – print the number of words n -l – print the number of lines With no options, all three are printed 9

Lecture Overview Basic text processing commands n head, tail, wc Redirection and pipes Getting to know vi Basic vi commands Advanced vi commands 10

Input and Output Redirection Normally, when you run a command, it gets input from the keyboard (stdin), and sends output to the screen (stdout) n Errors are sent to a third stream (stderr), which by default also points to the screen Input and output (as well as errors) can be read from or written to a file, by using the redirection meta-characters: '<' and '>' 11

Output Redirection We use the '>' operator to redirect the output of a command For example, to create a file containing a listing of '. c' files in the current directory: ls *. c > c_files. txt cat !$ boxes. c parse_tools. c shape. c tools. c 12

Input Redirection We use the '<' operator to redirect the input of a command For example, to see only the last two lines in the file 'c_files. txt': tail -2 < c_files. txt shape. c tools. c 13

Input / Output Redirection Input and output redirection can be combined in a single command For example: cat -n < my_phones. txt > numbered. txt head -4 !$ head -4 1 2 3 4 numbered. txt ADAMS, Andrew 7583 BARRETT, Bruce 6466 BAYES, Ryan 6585 BECK, Bill 6346 14

Appending Output When redirecting output to a file, the specified file is either created (if it does not exist), or overwritten (if it exists) The '>>' operator tells the shell to append the output to the file, without overwriting it To redirect stderr in addition to stdout, we use the operator '>&' 15

Appending Output – Example echo first line > lines. txt echo second line >> lines. txt echo third line >> lines. txt cat lines. txt first line second line third line echo fourth line > lines. txt cat lines. txt fourth line 16

Forced Redirection Output redirection fails if we try to: n Overwrite an existing file n Append to a non-existent file This behavior can be overridden by using the '!' modifier When '!' is added to a redirection operator, the redirection is forced to succeed 17

Forced Redirection – Examples The following will create the file 'lines. txt' if it does not exist, or overwrite it if it does: echo first line >! lines. txt This will also create 'lines. txt' if it does not exist, but append to it if it does echo first line >>! lines. txt 18

Pipes The following commands count how many '. c' files are in the current directory ls *. c > c_files. txt wc -l < c_files. txt This is not very efficient, and also leaves behind a temporary file as a side effect What we really want is to link the output of ls directly to the input of wc 19

Pipes A pipe takes the output of one command, and passes it as input to another Program 1 (ls *. c) stdout (or temp file) Pipe Program 2 (wc –l) stdin (or temp file) 20

Pipes The symbol for a pipe is a vertical bar – '|' The format of a command line which uses a pipe is: command 1 [args] | command 2 [args] Now the two commands from the previous example can be simply combined: ls *. c | wc -l 21

Pipes The use of pipes is actually a simple and easy form of inter-process communication Pipes can also be chained, so complex utilities can be created by combining simple building blocks When data passes through several pipes, these are sometimes referred to as filters 22

Pipes – Examples Print lines 6 and 7 of 'my_phones. txt': head -7 my_phones. txt | tail -2 GRAHAM, Linda 6141 HARMER, Peter 7484 Another way to get the same results: tail +6 my_phones. txt | head -2 To find the most recently changed file: ls -t | head -1 23

Lecture Overview Basic text processing commands n head, tail, wc Redirection and pipes Getting to know vi Basic vi commands Advanced vi commands 24

Editing Files With vi vi is the standard text editor in UNIX Unlike most editors or word processors that you have encountered, it is modal – the same input produces different results in different modes n Insert mode is for inserting text n Command mode is for everything else 25

Editing Files With vi has hundreds of commands, whose names require memorization It does not have menus or a GUI All of these characteristics make vi: n Very uncomfortable for beginners, but – n Very powerful for experts 26

Starting With vi To edit a file, type: vi [options] [file] You are now in command mode, so you cannot insert any text! To enter insert mode, type: 'i' Now you can type in text as with any other editor or word processor 27

Basic vi Commands To switch back to command mode, press Escape (this has no effect if you are already in command mode) In command mode, you can move around using the arrow keys, or these keys: 'h', 'j', 'k', 'l' To delete a character, type 'x' 28

Terminating a vi Session To exit vi, you need to switch to a third mode, called last line mode, by typing ': ' Then, type one of the following: n w – to save your changes n q – to quit an unmodified file n wq – to save your changes and quit n q! – to quit without saving your changes 29

Learning vi The best way to learn vi is: n n Start using it with a few basic commands Use available help sources to learn more features as you need them Getting help: n man vi n Typing ': help' within vi n UNIX books, the internet, etc. 30

Lecture Overview Basic text processing commands n head, tail, wc Redirection and pipes Getting to know vi Basic vi commands Advanced vi commands 31

vi Command Types And now for a more detailed look vi commands can be divided into groups: n Commands for entering insert mode n Commands for moving the cursor n Commands for deleting or changing text n Commands for searching or replacing text n Miscellaneous commands 32

Ways to Enter Insert Mode Command Description i Insert before cursor I Insert at start of line a Append after cursor A Append to end of line o Open a line below current line O Open a line above current line 33

Moving the Cursor Command Description h, j, k, l Left, down, up, right Arrow keys Left, down, up, right w Forward to start of next word b Backward to start of previous word Ctrl-F Forward one screen Ctrl-B Backward one screen 34

Moving the Cursor Command Description Ctrl-U Forward half a screen Ctrl-D Backward half a screen Ctrl-E Shift text down one line in window (cursor stays on same line) Ctrl-Y Shift text up one line in window (cursor stays on same line) % Move to the "mate" of this parenthesis or bracket 35

Moving the Cursor Command Description $ Move to end of current line 0 Move to start of current line ^ Move to first non-whitespace character in current line n. G, : n Go to line n gg Go to start of file G Go to end of file 36

Deleting Text Command Description x Delete character under cursor X Delete character before cursor dd Delete entire line dw Delete current word D Delete until end of line 37

Changing Text Command Description cc Change entire line cw Change current word C Change until end of line r Replace character under cursor R Start overwriting until ESC s Change one character and insert 38

Copy and Paste Command Description yy Yank (copy) current line yw Yank current word p Put after (or below) cursor P (capital) Put before (or above) cursor ]p Put, with indent adjusted 39

Lecture Overview Basic text processing commands n head, tail, wc Redirection and pipes Getting to know vi Basic vi commands Advanced vi commands 40

Defining Range Using Motion With some commands, it is possible to add a motion modifier which defines the range that they effect The general format is: command[motion] where motion is any of the available motion commands (e. g. d$) 41

Defining Range Using Motion In the command dw, d is the command (delete), and w is the motion (word) n dd (and similarly, cc) is a special case – operates on whole lines Examples: c$ Change until end of line d. G Delete until end of file y% Yank block (from current to pair bracket) 42

Repeating Commands Most vi commands can be repeated, using the following format: [repeat]command repeat defines how many times the given command should be executed repeat and motion can be combined in a single command (e. g. 3 dh) 43

Combining Repetition and Motion – Examples 5 dd Delete 5 lines, starting from current 3 x Delete 3 characters, starting from current 2 w Move forward 2 words 3 dw Delete 3 words, starting from current 4 yw Yank 4 words, starting from current y 4 w Same as above Note: In 4 yw, 4 is the repetition; in y 4 w, it is part of the motion 44

Searching for Patterns Command Description /pattern Search forward ? pattern Search backward n Repeat previous search N Repeat previous search in reverse direction pattern can also be a regular expression 45

Replacing Text The following command can be used to search and replace a pattern : s/search_string/replace_string/[g] This searches from the current position The g flag defines whether to replace only the first occurrence, or all matches pattern can also be a regular expression 46

Additional Commands Command Description . Repeat previous command u Undo last command U Undo all changes on current line Ctrl-R Redo last undo ~ Switch case and advance cursor J Merge current and next lines 47

Additional Commands Command Description >> Indent line one tab-stop << Un-indent line one tab-stop '' Go back to position before last move ma Mark current position as a 'a Jump to position marked as a 48

Commands in Last Line Mode Command Description : r file Insert file in cursor position : w file Write current file with new name : f Display details of current file : !command Run a shell command : r !command Run command, and insert result : set var Set a vi environment variable 49

Last Line Mode – Examples : !pwd Show current directory : !ls List contents of current directory : r !date Insert current date in cursor position : set nu Display line numbers : set nonu Do not display line numbers : set ai Auto-indent new lines according to indentation of previous ones : set noai Do not auto-indent new lines 50

The vi Startup File –. exrc vi provides various customization options Whenever vi is started, it first reads a file named. exrc from your home directory This file may contain any last line mode command, such as: n set – to set vi environment variables n map – to define new behavior for keys 51

A Sample. exrc File set set set autoindent number tabstop=4 shiftwidth=4 expandtab showmatch map ^ : r ~/template/header. txt^M map ^_ : r !date +%D^M map #6 : r ~/template/header. txt^M/Date^M: r !date +"%b %d, %Y"^Mk. Jjjj 0 map #9 : w!%^M: !chmod u+x %^M: !%^M 52