CPSC 325 Compiler Tutorial 2 Scanner Lex Tokens

CPSC 325 - Compiler Tutorial 2 Scanner & Lex

Tokens Input l Token Stream: Each significant lexical chunk of the program is represented by a token – – Operators & Punctuation: { } ! + - = * ; : … Keywords: if while return goto Identifier: id & actual name Constants: kind & value; int, floating-point character, string, …

Token – example 1 l Input text if( x >= y ) y = 10; l Token Stream IF LP ID(x) ID(y) GEQ Assign ID(y) INT(10) RP SEMI

Parser l Tokens IF LP ID(x) ID(y) GEQ Assign ID(y) INT(10) RP SEMI If. Stmt >= ID(x) assign ID(y) INT(10)

Sample Grammar l l l l Program : : = statement | program statement Statement : : = assign. Stmt | if. Stmt assign. Stmt : : = id = expr; if. Stmt : : = if ( expr ) Statement Expr : : = id | int | expr + expr id : : = a | b | … | y | z Int : : = 1 | 2 | … | 9 | 0

Why Separate the Scanner and Parser? l Simplicity & Separation of Concerns – – l Scanner hides details from parser (comments, whitespace, input files, etc. ) Parser is easier to build; has simpler input stream Efficiency – Scanner can use simpler, faster design l (But still often consumes a surprising amount of the compiler’s total execution time)

Principle of Longest Match l l In most of languages, the scanner should pick the longest possible string to make up the next token if there is a choice. Example return apple != banana; Should be recognized as 5 tokens return ID(apple) NEQ ID(banana) SEMI Not more (not parts of words or identifier, or ! And = as separate tokens)

Scanner DFA Example (1) White space or comments 0 Accept EOF end of input 1 Accept LP ( 2 ) ; Accept RP 3 4 Accept SEMI

Scanner DFA Example (2) White space or comments ! Accept NEQ = 6 5 other < 8 Accept NOT 7 = Accept LEQ 9 other 10 Accept LESS

Scanner DFA Example (3) White space or comments [0 -9] 11 Accept INT other 12

Scanner DFA Example (4) White space or comments [a-z. A-Z] 13 other 14 Accept ID or keyword

Lex/Flex l l l Use Flex instead of Lex Use Bison instead of yacc When compile, link to the library l l l flex file. lex gcc –o object lex. yy. c –ll object

Lex - Structure Declarations/Definitions %% l Rules/Production - Lex expression - white space - C statement (optional) %% l Additional Code/Subroutines l

Lex – Basic operators l l l l l *. . * | + ? [] - zero or more occurrences - “ANY” character - matches any sequence - separator - one or more occurrences. (a+ : == aa*) - zero or one of something. (b? : == (b+null) - choice, so [12345] (1|2|3|4|5) (Note: [*+] represent a choice between star and plus. They lost their specialty. - [a-z. A-Z] a to z and A to Z, all the letters. - * matches *, and . Match period or decimal point.