CSE P 501 Compiler Construction Scanner Regex Automata

CSE P 501 – Compiler Construction Scanner Regex Automata Hand-Written Scanner Grammars & BNF Next Spring 2014 Jim Hogg - UW - CSE P 501 B-1

Scanner Source Front End chars ‘Middle End’ Back End IR IR Select Instructions Scan tokens Parse IR Optimize Allocate Registers IR AST Emit Semantics IR Target IR Machine Code AST = Abstract Syntax Tree IR = Intermediate Representation Spring 2014 Jim Hogg - UW - CSE P 501 A-2

Automatic or Hand-Written? n Use a scanner-generator - JFlex regex define tokens JFlex . jflex Scanner. java OR n Write a scanner, in Java, by hand n n Easy and enlightening Will see an outline of how, later Spring 2014 Jim Hogg - UW - CSE P 501 B-3

Reminder: a token is. . . class C { public int fac(int n) { // factorial int nn; if (n < 1) nn = 1; else nn = n * this. fac(n-1); return nn; } } Key for Char Stream: ◊ ∙ newline n space class∙C∙{◊∙∙public∙int∙fac(int∙n)∙{∙∙//∙factorial◊∙∙∙∙int∙nn; ◊∙∙∙∙if(n∙<∙ 1 )◊∙∙∙∙∙∙nn∙=∙ 1; ◊∙∙∙∙else◊∙∙∙∙nn∙=∙n∙*∙(this. fac(n 1)); ◊∙∙∙∙return∙nn; ◊∙∙}◊} CLASS ID: C LBRACE PUBLIC INT ID: fac LPAREN INT ID: n RPAREN LBRACE INT ID: nn SEMI IF LPAREN ID: n LT ILIT: 1 RPAREN ID: nn EQ ILIT: 1 ELSE ID: nn EQ ID: n TIMES LPAREN ID: this DOT ID: fac LPAREN ID: n MINUS ILIT: 1 RPAREN SEMI RETURN ID: nn SEMI RBRACE Spring 2014 Jim Hogg - UW - CSE P 501 A-4

A Token in your Java scanner class Token { public int kind; public int line; public int column; public String lexeme; public int value; } // // // eg: LPAREN, ID, ILIT for debugging/diagnostics eg: “x”, “Total”, “(“, “ 42” attribute of ILIT Obviously this Token is wasteful of memory: • lexeme is not required for primitive tokens, such as LPAREN, RBRACE, et • value is only required for ILIT But, there's only 1 token alive at any instant during parsing, so no point refining into 3 leaner variants! Spring 2014 Jim Hogg - UW - CSE P 501 B-5

Typical Tokens n Operators & Punctuation n ] ; : if while for goto return switch void … Identifiers n n ( Keywords n n Single chars: + * = / Double chars: : : <= == != A single ID token kind, parameterized by lexeme Integer constants n A single ILIT token kind, parameterized by int value See jflex-1. 5. 0examplesjava. flex for real example Spring 2014 Jim Hogg - UW - CSE P 501 B-6

Token Spotting if(a<=3)++grades[1]; // what are the tokens? (no spaces) public int fac(int n) { // what are the tokens? (need spaces? ) Counter-example: fixed-format FORTRAN: DO 50 I = 1, 99 DO 50 I = 1. 2 Spring 2014 // DO loop // assignment: DO 50 I = 1. 2 Jim Hogg - UW - CSE P 501 B-7

Principle of Longest Match n n Scanner should pick the longest possible string to make up the next token (“greedy” algorithm) Example return idx <= iffy; should be scanned into 5 tokens: RETURN n n ID: idx LEQ ID: iffy SEMI <= is one token, not two iffy is an ID, not IF followed by ID: fy Spring 2014 Jim Hogg - UW - CSE P 501 B-8

Regex n The syntax, of most programming languages can be specified using Regular Expressions n n n “REs” in Cooper&Torczon “regex” is more common Tokens can be recognized by a deterministic finite automaton (DFA) n DFA (a Java class) is almost always generated from regex using a software tool, such as JFlex Spring 2014 Jim Hogg - UW - CSE P 501 B-9

Regex Cheat Sheet Pattern Matches? a a a* zero or more a’s a+ one or more a’s a? zero or one a a|b a or b ab a followed by b Pattern Matches? [c-f] one of c or d or e or f [^0 -3] any one character except 0 -3 . any character, except newline Precedence: * (highest), concatenation, | (lowest) Parentheses can be used to group regexs as needed Notice meta-characters, in red Escaped characters: * + ? | . t n Spring 2014 Jim Hogg - UW - CSE P 501 B-10

Regex Examples regex Meaning? [abc]+ [abc]* (Kleene closure) [0 -9]+ [1 -9][0 -9]* [a-z. A-Z_][a-z. A-Z 0 -9_]* (0|1)* 0 (a|b)*aa(a|b)* Check free online Regex tutorials if you are rusty. Eg: http: //regexone. com/ Experiment with a regex-capable editor. Spring 2014 Eg: http: //www. editpadpro. com/ Jim Hogg - UW - CSE P 501 B-11

regex n Defined over some alphabet Σ n n For programming languages, alphabet is ASCII or Unicode If re is a regular expression, L(re ) is the language (set of strings) generated by re Spring 2014 Jim Hogg - UW - CSE P 501 B-12

regex macros n Possible syntax for numeric constants Digit = [0 -9] Digits = Digit+ Number = Digits (. Digits )? ( [e. E] (+ | -)? Digits ) ? n How would you describe this set in English? n What are some examples of legal constants (strings) generated by Number? n Tools like JFlex accept these convenient macros Spring 2014 Jim Hogg - UW - CSE P 501 B-13

Automata n n Finite automata (state machines) can be used to recognize strings generated by regular expressions Can build automaton by-hand or automagically n n Will not build by-hand in this course Will use the JFlex tool: given a set of regex, it generates an automaton recognizer (a Java class) Spring 2014 Jim Hogg - UW - CSE P 501 B-14

Finite Automata Terminology Phrase Abbreviation Finite Automaton FA Deterministic Finite Automaton DFA Non-deterministic Finite Automaton NFA Finite-State Automaton FSA = {DFA, NFA} Spring 2014 Jim Hogg - UW - CSE P 501 B-15

DFA for “cat” regex = cat c a Accepting State (double circles) Start State Spring 2014 t Jim Hogg - UW - CSE P 501 B-16

DFA for ILIT regex = [0 -9]* = [0 -9]+ 0 -9 1 2 We have labelled the states Spring 2014 Jim Hogg - UW - CSE P 501 B-17

DFA for ID regex = [a-z. A-Z_][a-z. A-Z 0 -9_]* _ a-z 0 Spring 2014 _ a-z A-Z 1 Jim Hogg - UW - CSE P 501 0 -9 B-18

DFAs work like this. . . 1. scan the input text string, character-by-character 2. following the arc/edge corresponding to the character just read 3. if there is no arc for the character just read, then, either: a. b. if you are in an accepting state: you're done. Success! if you are not in an accepting state: you're done. Failure! Spring 2014 Jim Hogg - UW - CSE P 501 B-19

DFAs work like this - examples 1. 2. 3. Scan "fac(int n); " for the regex, alphaid = [a-z]+ (lower-case alphas) We hit "(" and are already in state 1. Success Scan "23; " for regex alphaid There is no arc for "2". We are still in state 0. Failure Scan "today" for regex alphaid We hit end-of-string and are already in state 1. Success 0 a-z 1 Note: no need to add arcs to the DFA for all error cases - they are implicit Spring 2014 Jim Hogg - UW - CSE P 501 B-20

Thompson’s Construction: Combining DFAs b a DFA for: a a DFA for: b b ε ε a NFA for: ab ε NFA for a|b ε Spring 2014 b ε Jim Hogg - UW - CSE P 501 B-21

Combining DFAs, cont’d b a DFA for: b ε ε a ε NFA for: a* ε Spring 2014 Jim Hogg - UW - CSE P 501 B-22

Exercise n Draw the NFA for: b(at|ag) | bug b b Spring 2014 a t a g u Jim Hogg - UW - CSE P 501 g B-23

Exercise n Draw the NFA for: b(at|ag) | bug a t a g b b Spring 2014 u Jim Hogg - UW - CSE P 501 g B-24

NFA for a(b|c)* a b c To recognize "acb" successfully, we need to: • • • guess the future correctly backtrack and retry if we fail to recognize somehow execute all possible paths None of these is attractive! Can we construct an equivalent DFA? b a c Spring 2014 Jim Hogg - UW - CSE P 501 B-25

Finite State Automaton (FSA) n A finite set of states n n A set of transitions from state to state n n Each labeled with symbol from Σ, or ε Operate by reading input symbols (usually characters) n n n One marked as initial state One or more marked as final states States sometimes labeled or numbered Transition can be taken if labeled with current symbol ε-transition can be taken at any time (free bus ride) Accept when final state reached & no more input n n Scanner uses an FSA as a subroutine – accept longest match from current location each time called, even if more input Reject if no transition possible, or no more input and not in final state (DFA) Spring 2014 Jim Hogg - UW - CSE P 501 B-26

DFA vs NFA n Deterministic Finite Automata (DFA) n n No choice of which transition to take In particular, no ε transitions No guessing Non-deterministic Finite Automata (NFA) n n Choice of transition in at least one case Accepts if some way to reach final state on given input Reject if no possible way to final state How to implement in software? Spring 2014 Jim Hogg - UW - CSE P 501 B-27

DFAs in Scanners n We really want DFA for speed: no backtracking, no guessing, no foretelling the future n Conversion from regex to NFA is easy, right? n But how to turn an NFA into an equivalent DFA? n Turns out to be obvious (once seen) and easy Spring 2014 Jim Hogg - UW - CSE P 501 B-28

NFA to DFA NFA for a(b|c)* 4 0 a 1 2 b 5 8 3 6 c 9 7 Starting with the above NFA, we want to 'collapse' epsilon edges, ending up with a DFA that recognizes, and rejects, the same char strings. Ideally, we will end up with: b a 1 0 c Spring 2014 Jim Hogg - UW - CSE P 501 B-29

NFA to DFA NFA for a(b|c)* 4 0 a 1 2 • 5 8 3 6 • b c 9 7 Begin in the Start state Foreach labelled arc leaving that state, what set of states can I reach, along labelled arc, or along transitions? Spring 2014 Jim Hogg - UW - CSE P 501 B-30

NFA to DFA n 4 n 0 a n 1 n 2 b NFA for a(b|c)* n 5 n 8 n 3 n 6 c n 9 n 7 NFA State a b c d 0 = n 0 d 1 = {1, 2, 3, 4, 6, 9} none d 2 = {3, 4, 5, 6, 8, 9} d 3 = {3, 4, 6, 7, 8, 9} d 2 = {3, 4, 5, 6, 8, 9} none d 2 = {3, 4, 5, 6, 8, 9} d 3 = {3, 4, 6, 7, 8, 9} Spring 2014 Jim Hogg - UW - CSE P 501 B-31

NFA to DFA for a(b|c)* d 2 b d 0 a b c d 1 c b c d 3 NFA State a b c d 0 d 1 - - d 1 - d 2 d 3 d 2 - d 2 d 3 Spring 2014 Jim Hogg - UW - CSE P 501 B-32

NFA to DFA - Even Better DFA for a(b|c)* d 0 a b d 1 c • Can reduce number of states further, to yield above result • If interested, see books for details • States minimization is not examined in P 501 Spring 2014 Jim Hogg - UW - CSE P 501 B-33

From NFA to DFA n Subset construction (equivalence class) n n Construct DFA from NFA, where each DFA state represents a set of NFA states Key idea n State of DFA after reading some input is the set of all states the NFA could have reached after reading the same input n Algorithm: example of a fixed-point computation n If NFA has n states, DFA has at most 2 n states n => DFA is finite, can construct in finite # steps Spring 2014 Jim Hogg - UW - CSE P 501 B-34

Build DFA for: b(at|ag) | bug from its NFA 2 0 b a 3 t 4 1 5 b 8 a 6 u 9 g 12 7 g 10 11 NFA State a b g t u d 0 = 0 - {1, 2, 5, 9} - - - d 1 = {1, 2, 5, 9} ? ? ? Spring 2014 Jim Hogg - UW - CSE P 501 B-35

Build DFA for: b(at|ag) | bug from its NFA 2 b 0 a 3 t 4 1 5 8 b a 6 u 9 g 12 7 g 10 11 NFA State a b g t u d 0={0} - d 1={1, 2, 5, 9} - - - d 1 = {1, 2, 5, 9} d 2={3, 6} - - - d 3={10} d 2 = {3, 6} - - d 4={7} d 5={4, 12} - d 3 = {10} - - d 6={11, 12} - - TBD ? ? ? Spring 2014 Jim Hogg - UW - CSE P 501 B-36

Hand-Written Scanner n Idea: show a hand-written DFA for some typical tokens n n Setting: Parser calls scanner whenever it wants next token n JFlex provides next_token n Then use to construct hand-written scanner Scanner stores current position in input For illustration only. Course project will use JFlex scanner-generator Note - most commercial compilers use hand-written scanners - generally faster Spring 2014 Jim Hogg - UW - CSE P 501 B-37

Scanner DFA Example – Part 1 whitespace or comments 0 end of input ( ) ; Spring 2014 1 Accept EOF 2 Accept LPAREN 3 Accept RPAREN 4 Accept SEMI Jim Hogg - UW - CSE P 501 B-38

Scanner DFA Example – Part 2 ! 5 = [other ] < 8 = [other ] Spring 2014 6 Accept NEQ 7 Accept NOT 9 Accept LEQ 10 Accept LESS Jim Hogg - UW - CSE P 501 B-39

Scanner DFA Example – Part 3 [0 -9] 11 [0 -9] [other ] Spring 2014 12 Accept ILIT Jim Hogg - UW - CSE P 501 B-40

Scanner DFA Example – Part 4 [a-z. A-Z] 13 [a-z. A-Z 0 -9_] [other ] n 14 Accept ID or keyword Strategies for handling identifiers vs keywords n n Hand-written scanner: look up identifier-like things in table of keywords Machine-generated scanner: generate DFA with appropriate transitions to recognize keywords Spring 2014 Jim Hogg - UW - CSE P 501 B-41

Scanner – class, ctor, skip. White public class Scanner { private String prog; private int p; // the Mini. Java program to be scanned // index in 'prog' of current char public Scanner(String prog) { this. prog = prog; p = 0; } private void skip. White() { char c = prog. char. At(p); while ( Character. is. Whitespace(c) ) c = prog. char. At(++p); } Spring 2014 Jim Hogg - UW - CSE P 501 B-42

Scanner- id private Token id() { int p. Begin = p; char c = prog. char. At(p); // remember begin index of id // current char - alphabetic while ( Character. is. Alphabetic(c) || Character. is. Digit(c) || c == '_') { c = prog. char. At(++p); } return new Token(ID, prog. substring(p. Begin, p)); } Spring 2014 Jim Hogg - UW - CSE P 501 B-43

Scanner - i. Lit private Token i. Lit() { int p. Begin = p; // remember begin index of lexeme char c = prog. char. At(p); // current char int val = Character. get. Numeric. Value(c); // convert to int while ( Character. is. Digit(c) ) { // step thru chars of number c = prog. char. At(++p); val = 10 * val + Character. get. Numeric. Value(c); } String lex = prog. substring(p. Begin, p); return new Token(ID, lex, val); } Spring 2014 Jim Hogg - UW - CSE P 501 B-44

Scanner - next. Token public Token next. Token() { skip. Whitespace(); char c = prog. char. At(p); char n = prog. char. At(p + 1); // returns at prog[p] // current char in 'prog' // next char in 'prog' switch (c) { case ‘>': if (n == '=') { p++; return new Token(GEQ, “>="); } else { p++; return new Token(GT, “>"); } //. . . case '+': p++; return new Token(PLUS, "+"); //. . . } // end of switch Spring 2014 Jim Hogg - UW - CSE P 501 B-45

Scanner – next. Token, cont’d if (Character. is. Digit(c)) { return this. i. Lit(); } else if (Character. is. Alphabetic(c)) { return this. id(); } else { return new Token(BAD, ""); } } // end of next. Token } // end of class Scanner An entire hand-written scanner for Mini. Java takes ~100 lines of Java Spring 2014 Jim Hogg - UW - CSE P 501 B-46

Grammars & BNF n Since the 60 s, the syntax of every significant programming language has been specified by a formal grammar n n First done in 1959 with BNF (Backus-Naur Form); used to specify ALGOL 60 syntax Borrowed from the linguistics community (Noam Chomsky) Spring 2014 Jim Hogg - UW - CSE P 501 B-47

Grammar for a Tiny Language n n n n program statement assign. Stmt if. Stmt expr id ilit statement | program statement assign. Stmt | if. Stmt id = expr ; if ( expr ) statement id | ilit | expr + expr a|b|c|i|j|k|n|x|y|z 0|1|2|3|4|5|6|7|8|9 Note: often see : : = used instead of Spring 2014 Jim Hogg - UW - CSE P 501 B-48

program : : = statement | program statement : : = assign. Stmt | if. Stmt assign. Stmt : : = id = expr ; if. Stmt : : = if ( expr ) statement expr : : = id | ilit | expr + expr id : : = a | b | c | i | j | k | n | x | y | z ilit : : = 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 Example Derivation P S A I E id ilit a = 1 ; if ( a Spring 2014 + 1 ) Jim Hogg - UW - CSE P 501 S | P S A | I id = E ; if ( E ) S id | ilit | E + E [a-z] [0 -9] b = 2 ; B-49

Parse Tree - First Few Steps P P S A I E id ilit + 1 ) b = S S | P S A | I id = E ; if ( E ) S id | ilit | E + E [a-z] [0 -9] A id = E ; ilit a = 1 ; if ( a 2 ; B-50

Parse Tree - Complete P S A I E id ilit P P S S I A id = if E ilit ; ( E ) E + E id ilit S | P S A | I id = E ; if ( E ) S id | ilit | E + E [a-z] [0 -9] S A id = E ; ilit a = 1 ; if ( a + 1 ) b = 2 ; B-51

Alternative Notations n There are several syntax notations for productions in common use; all mean the same thing if. Stmt : : = if ( expr ) statement if. Stmt if ( expr ) statement <if. Stmt> : : = if ( <expr> ) <statement> Spring 2014 Jim Hogg - UW - CSE P 501 B-52

Formal Languages & Automata Theory n Alphabet: a finite set of symbols ( eg: [a-z. A-Z 0 -9_] ) n String: a finite, possibly empty sequence of symbols from an alphabet n Language: a set, often infinite, of strings n Finite specifications of (possibly infinite) languages n Grammar – a generator; a system for producing all strings in the language (and no other strings) n A particular language may be specified by many different grammars n A grammar specifies only one language Spring 2014 Jim Hogg - UW - CSE P 501 B-53

Productions n The rules of a grammar are called productions n Rules contain n Meaning of n n Nonterminal symbols: grammar variables (program, statement, id, etc) Terminal symbols: concrete syntax that appears in programs (a, b, c, 0, 1, if, (, ), … ) nonterminal <sequence of terminals and non-terminals> In a derivation, an instance of non-terminal can be replaced by the sequence of terminals and non-terminals on its RHS Often, there are two or more productions for one nonterminal – use any in different parts of derivation Spring 2014 Jim Hogg - UW - CSE P 501 B-54

Two ways to Parse n n Parse: re-construct the derivation (syntactic structure) of a program More prosaically: fill the gap between top and bottom of page with a parse tree: n n Start at top; build tree downwards, sweeping left-to-right. This is called a "top-down" parse. What we just did for the "Tiny Language" example Start at bottom; build little trees that join upwards. Called a "bottom -up" parse. What CUP does for us. Spring 2014 Jim Hogg - UW - CSE P 501 B-55

Why Separate Scanner and Parser? n n In principle, a single recognizer could work directly from a concrete, character-by-character grammar In practice this is never done: always scan chars to tokens, because: n Simplicity & Separation of Concerns n n n Scanner hides details from parser (comments, whitespace, input files, etc) Parser becomes easier to build; has simpler input - stream-of-tokens Efficiency n n Scanner can use simpler, fast design But still often consumes a surprising amount of the compiler’s total execution time - it touches every char in source program Spring 2014 Jim Hogg - UW - CSE P 501 B-56

Project Notes n For Mini. Java project n n n Use JFlex scanner-generator tool Use CUP parser-generator tool The two work together CUP generates a file of token kinds into sym. java (SEMI = 28, LT = 18, etc) JFlex needs these definitions. To bootstrap this process, inspect the Mini. Java grammar and devise your own set of token kinds See Mini. Java page at: http: //www. cambridge. org/resources/052182060 X/ Spring 2014 Jim Hogg - UW - CSE P 501 B-57

Next n Homework: paper exercises on regex and FAs n Next week: first part of the compiler assignment – the scanner n n Send partner info to Nat if you want project space Next topic: parsing n n Will do LR parsing first, for the project (CUP) Cooper&Torczon chapter 3 Spring 2014 Jim Hogg - UW - CSE P 501 B-58