Regular Expressions Austin Linux Meetup Some people when

Regular Expressions Austin Linux Meetup

Some people, when confronted with a problem, think “I know, I’ll use regular expressions. ” Now they have two problems. - Jamie Zawinski

What are Regular Expressions? A concise grammar for doing pattern matching in strings or text. Often abbreviated as Reg. Ex.

Why Learn Reg. Ex? If you use Linux, regular expressions are everywhere!

Linux System Administration and Daily Usage • • • grep/egrep: find strings in text files that match a pattern. sed: find pattern in text and filter or transform into something else. find: find files or directories awk: data extraction and reporting tool bash: shell scripting ls: list files

Linux Server Administration Every component of LAMP stack uses Reg. Ex. • • • Apache NGINX My. SQL PHP Perl Python

Apache mod_rewrite Rewrite. Cond %{HTTP_HOST} (. *) Rewrite. Rule ^/(. *) /sites/%1/$1

NGINX Http. Rewrite. Module if ($http_user_agent ~ MSIE) { rewrite ^(. *)$ /msie/$1 break; }

My. SQL SELECT * FROM `table` WHERE column REGEXP '^Colou? r'

Programming Languages All programming languages have some form of Reg. Ex support Perl: if ($var =~ m/regex/) { do something; } PHP: if (preg_match("/regex/u", $var_to_match_against)) { do something; } Java: Pattern r = Pattern. compile(“regex”); Matcher m = r. matcher(line); if (m. find() { do something; } Python, Javascript, C#, C++, Visual Basic, Ruby, etc.

Linux Utilities nano • • VIM Emacs nmap Postfix (Open|Libre)Office Geany Sublime Text Kate

Reg. Ex Basics. Match any one character ^ Start of line $ End of line

Reg. Ex Basic Examples bag the string 'bag' ^bag 'bag' at the beginning of a line. bag$ 'bag' at the end of a line. ^bag$ 'bag' is the only word on the line b. g second letter is any single character ^. . . $ any line containing exactly three characters

Reg. Ex Basics [ ] Character class [^ ] Negated character class

Reg. Ex Basic Examples b[aeiou]g second letter is a lowercase vowel b[^aeiou]g second letter is not a lowercase vowel [a-z] Lowercase alphabet characters [^0 -9 A-Za-z] Any non-alphabet or number: symbols and spaces

Reg. Ex Basics | Alternation ? Optional * Zero or more (greedy) + One or more (greedy)

Reg. Ex Basic Examples Color|Colour Color or Colou? r Color or Colour 10* 1, 100, 1000, … 10+ 10, 1000, 10000

Reg. Ex Basics *? Zero or more (lazy) +? One or more (lazy) {min, max} Range

Reg. Ex Basic Examples <li>item 1</li><li>item 2</li> <li>. *</li> <li>item 1</li><li>item 2</li> <li>. *? </li> <li>item 1</li> A{3, 5} A{4} AAA, AAAA

Reg. Ex Basics n, t, Escape characters s, d, w Class shorthands (space, digit, word) S, D, W Class negative shorthands b, <, > match word boundary positions

Reg. Ex Basics ( ) Grouping and capturing 1, 2 Back references

Reg. Ex Basic Examples (happy) 1 (joy) 2 happy joy ^s+$ Blank lines with spaces

Demo Examples ls -l [a-z] * ls -l [^a-z]* Find a string in double quotes grep '"[^”]*"' file. txt Find uncommented lines in config file grep '^[^#]' file

Demo Examples Duplicate words egrep -i 'b([a-z]+)s+1b' file. txt 12 -hour time egrep -i 'b(1[012]|[1 -9]): [0 -5][0 -9]b (am|pm)' file. txt 24 -hour time egrep -i 'b([01]? [0 -9]|2[0 -3]): [0 -5][0 -9]b (am|pm)' file. txt

Demo Examples Email address egrep -i ‘b[A-Z 0 -9. _%+-]+@[A-Z 0 -9. -]+. [A-Z]{2, 4}b’ file. txt IP Address Wrong: egrep ‘[0 -9]{1, 3}. [0 -9]{1, 3}’ file. txt Good: egrep ‘([01]? dd? |2[0 -4]d|25[0 -5]). ([01]? dd? |2[0 -4]d|25[0 -5])’ file. txt

Demo Examples HTML Tags Good: egrep '<[^>]+>' file. txt Better: egrep <("[^"]*"|'[^’]*'|[^'">])*> Split parent directory and current directory pwd | sed -r "s/^(. *)/(. *)$/The current directory is 2 and is child of 1/"

Demo Examples Change date format from MM-DD-YYYY to YYYY-MM-DD cat file. txt | sed -r 's/([0 -9]{2})-([09]{4})/3 -1 -2/g'

Advanced Reg. Ex - Character Class • [[: alpha: ]] • [[: alnum: ]] • [[: cntrl: ]] • [[: lower: ]] • [[: upper: ]]

Advanced Reg. Ex - Unicode Block Character Properties • • • p{Lowercase_Letter} p{Uppercase_Letter} p{Symbol} p{Currency_Symbol} p{Punctuation}

Advanced Reg. Ex - Unicode Block Properties • • • p{Greek} p{Arabic} p{Han} p{Hiragana} p{Katakana} p{Hangul}

Advanced Reg. Ex - Lookarounds (? <= ) Positive lookbehind (? <! ) Negative lookbehind (? = ) Positive lookahead (? ! ) Negative lookahead

Advanced Reg. Ex Examples - Perl Find all lines with Japanese text if ($line =~ m/p{Han}|p{Hiragana}|p{Katakana}/) { say $line; } Find all lines that don’t have Japanese text if ($line =~ m/P{Han}|P{Hiragana}|P{Katakana}/) { say $line; }

Advanced Reg. Ex Examples Add commas to number strings. Example: 1234567890 becomes 1, 234, 567, 890 s/(? <=d)(? =(ddd)+(? !d))/, /g

Reg. Ex Resources Reg. Exr - Tool for testing and learning Reg. Ex http: //gskinner. com/Reg. Exr/ Reg. Ex Tutorial site http: //www. regular-expressions. info Excellent regex resource: http: //linuxreviews. org/beginner/tao_of_regular_expressions/ Reg. Ex http: //gnosis. cx/publish/programming/regular_expressions. html http: //www. grymoire. com/Unix/Regular. html Regex Coach Regex Buddy: good source http: //www. regular-expressions. info/email. html http: //www. regular-expressions. info/reference. html