Unix Utilities SED Basics M Raghavan alias Saravanan

Unix Utilities – SED Basics -M Raghavan alias Saravanan 08 Jan 2011 Saturday

SED Introduction SED – Stream Editor Descendant of the popular Unix editor ‘ed’ Though ‘ed’ not in regular use, gets shipped with all *nix OSes Available in all *nix OS (including Macintosh) by default Generally found in /bin/sed or /usr/bin/sed

SED Installation If not available by default, you can get it from GNU website -> ftp: //ftp. gnu. org/pub/gnu/sed/. The file name would be : sed-<version>. tar Steps to proceed tar –zxf sed-4. 1. 2. tar [Version may vary!] cd sed-4. 1. 2 sh. /bootstrap. h [Need to revisit the manuals] Verify the presence of sed --version

SED Fundamentals A streaming editor – Imagine a stream of water plus pipe for flow Doing the editing in the stream without reflecting the changes outside Uses something called a Pattern space or Pattern buffer to manipulate the contents Works line by line! – no matter how much you like to do! Does NOT alter the input file Generally prints the output in the console You can redirect it to any other resource of your choice

Invoking/Using Sed Just by invoking the sed editor does NOT mean that it would so some job Usually instructions are passed in the form of ‘editing commands’ which are single characters ‘d’ – for deletion ‘p’ – for printing ‘s’ – for substitution etc. , Sed needs the input to be passed via any of the 2 ways Pipe -> [cat /etc/passwd | sed ‘d’] File Input -> [sed –e ‘d’ /etc/passwd]

Sample Sed command – ‘d’ cat /etc/passwd | sed ‘d’ The editing command ‘d’ is meant for deletion The editing commands are supplied between the pair of single quotes Otherwise they may get parsed incorrectly by the shell thereby resulting in an unexpected output Working of this command The output of the file /etc/passwd is supplied (piped) through pipe to sed The Sed is instructed to delete the line and print the result to console Repeats for each line by taking one line into its pattern space, does it job (delete) and print the result of pattern buffer Here after deletion, it finds nothing in the buffer , hence nothing to print

Sample Command – ‘e’ sed –e ‘d’ /usr/passwd Same as above but the input is provided through a file rather than the data piped sed –e ‘d’ /usr/passwd > /tmp/newpasswd Redirecting the output to a new file, which may be the most preferred option for the later usage! NOTE: Don’t redirect to the original input file unless you are sure of what sed produces as output!

Sample Command – ‘p’ sed –e ‘p’ /usr/passwd ‘p’ editing command prints the pattern space while editing Turned On by default You can turn it off by using any of the arguments given below ‘-n’ ‘-- quiet’ ‘-- silent’ Try the following cat /etc/passwd | sed ‘p’ | head -10 cat /etc/passwd | sed –n ‘p’ | head -10

SED Errors Easy to get an error! By incorrectly specifying the editing command Different GNU versions of SED has a different style of throwing errors Overall none of them user friendly and they are cryptic Note : Syntax requires attention in detail!

Address Simply a location in a file Range where a particular editing command should be applied Example: cat /etc/passwd | sed ‘ 1 d’ | more Deletes the 1 st line The address is specified through a numeric value that denotes the particular line numbers in the file to be effective The number just means that particular line number It does NOT mean the count (total # of lines) th

Address Range How to specify a range of addresses? Address Range comes to rescue in the form of [start. Number, end. Number]<editing. Command> start. Number, end. Number can be any number in chronological order Example: sed -e ‘ 1, 5 d’ /etc/passwd deletes the lines 1 to 5 What if you specify the address in Reverse Chronological Order? Like sed –e ‘ 10, 4 d’/etc/passwd ? Takes the lines from #1 into pattern space until it reaches #10 After finding the line #10, it deletes it Looks for line #9 -4 but does NOT do anything in backwise!

Address Range (Contd. , ) What if you specify a non-existing range of addresses? What if you miss out a number (ending) of the address range? Example : sed ‘ 1, d’ ? Sed gives you a cryptic error Unix box : [sed: -e expression #1, char 3: unexpected `, ’] My Macbook : [sed: 1: "1, d": expected context address] What if you miss out a number (starting) of the address range? Example : sed ‘, 5 d’ ? Sed thinks ‘, ’ as an editing command tells you no such cmd exists! Unix box : [sed: -e expression #1, char 1: unknown command: `, ’]

Address Range (Contd. , ) Dynamic appending of lines Possible through the “+” character before the address range number Example: cat /etc/passwd | sed ‘ 4, +5 d’ Starts from line #4 and then include the subsequent 5 lines

Address Negation What if you want to exclude a set of lines? Use the “exclamation symbol” (“!”) before the address number range Example: cat /etc/passwd | sed ‘ 1, 5!d’ Deletes everyline except the lines in the range (1 to 5)

Address Steps Way to specify the incremental operation Using the tilde symbol (“~”) Example: cat /etc/passwd | sed ‘ 1~3 d’ Start from line #1 and then skip 3 lines, delete the next line Deletes every third line alternatively starting from line #1

Substitution Very important feature Useful in our day to day life! Through the editing command ‘s’ Usage : Replacing strings in a file! Specify the source and target strings by bookending them with a forward slash (“/”) Example : cat /etc/passwd | sed ‘s/root/toor/’ Replaces the term ‘root’ with ‘toor’ Note: It replaces only the first occurence! What if you miss the terminating slash at the end? You would get the traditional error in Sed’s own way!

Substitution (Contd. , ) What if you need to replace the strings in a whole file? Use the character ‘g’ (global) ‘g’ is a substitution flag! Example: cat /etc/passwd | sed ‘s/root/toor/g’ The flags should be appended at the last, precisely after the last forward slash

Substitution Flags Few major flags g – global replace NUMBER – replaces only the NUMBERth occurence p – if substitution was made, print the pattern space I or i – to do it in a case insensitive manner w <file. Name> -- if the substitution was made, write the result into the <file. Name> Examples cat /etc/passwd | sed ‘s/Root/toor/ig’ cat /etc/passwd | sed ‘s/root/toor/3’ Replaces only the third occurrence of the term ‘root’ in the file

Alternate String Separator What if you want the forward slash “/” to be part of your input data? If you want to replace the home directory /root to /toor? Two ways to do Use an alternate String separator Escape the separator

Alternate String Separator (Contd. , ) Alternate String Separator – any other special character which can be replaced for “/” Example: cat /etc/passwd | sed ‘s: /root: /toor: ’ | head -2 Don’t forget the fact both the strings should be surrounded by the separators cat /etc/passwd | sed ‘s: /root: /toor’ – will give an error!

Alternate String separator (Contd. , ) Escaping the String separator is very handy but pay attention to it Use the backslash for every character/separator Example: cat /etc/passwd | sed ‘s//root//toor/’ | head -2 Each “/” is escaped with a “” as “/”

Replace Operations Replacing with an empty space cat /etc/passwd |sed ‘s/root//g’ | head -2 deletes/removes the word ‘root’ in entire file (‘g’) Replace works fine with the string with space as well cat /etc/passwd |sed ‘s/root user//g’ | head -2 Removes the word ‘root user’ in the entire file (‘g’)

Address Substituion Address can be used for Substitution as well (as like for deletion ‘d’) Example: cat /etc/passwd | sed ’ 10 s/sh/quiet/g’ Replaces the term ‘sh’ with ‘quiet’ only in the 10 th line cat /etc/passwd | sed ’ 1, 5 s/sh/quiet/g’ Replaces the term ‘sh’ with ‘quiet’ for the lines #1 to #5

SED Advanced Search/Replace operations in an entire file Using Regular Expressions. . Will be covered in next session. . .

Useful Resources http: //face. centosprime. com/macosxw/sed-one- liners/ http: //en. wikipedia. org/wiki/Sed GNU sed Manual : http: //www. gnu. org/software/sed/manual/sed. html Sed An Introduction and Tutorial : http: //www. grymoire. com/Unix/Sed. html (*) Beginning Shell Scripting Book from Wrox Publishers --- I am referring currently!

Queries ? ? ?

Thank you!!!