Lexical Analysis Equality vs Equivalence Are these regular

Lexical Analysis

Equality vs Equivalence • Are these regular expressions equal? R = a a* (b | c) S = a* a (c | b) . . . No! • Yet, they describe the same language. L(R) = L(S) • “Equivalence” of regular expressions If L(R) = L(S) then we say R S “R is equivalent to S” • From now on, we’ll just say R = S to mean R S

Algebraic Law of Regular Expressions

Regular Definition • If Σ is an alphabet of basic symbols then a regular definition is a sequence of the following form: d 1 r 1 d 2 r 2 ……. . dn rn where • Each di is a new symbol such that di Σ and di dj where j < I • Each ri is a regular expression over Σ {d 1, d 2, …, di-1)

Regular Definition

Addition Notation / Shorthand

Unsigned Number 1240, 39. 45, 6. 33 E 15, or 1. 578 E-41 digit 0 | 1 | 2 | … | 9 digits digit* optional_fraction . digits | optional_exponent ( E ( + | -| ) digits) | num digits optional_fraction optional_exponent Shorthand digit 0 | 1 | 2 | … | 9 digits digit+ optional_fraction (. digits ) ? optional_exponent ( E ( + | -) ? digits) ? num digits optional_fraction optional_exponent

Nonregular Sets

Example of Regular Expression for numbers digit 0|1|…|9 digits digit* optional_fraction . digits| optional_exponent ( E (+|-| ) digits ) | num digits optional_fraction optional_exponent Using shorthands: ? means “zero or one instance of” r? is a shorthand of r | digit 0|1|…|9 digits digit+ optional_fraction (. digits)? optional_exponent ( E (+|-| ) digits ) ? num digits optional_fraction optional_exponent

Transition Diagram • A stylized flowchart produced intermediately in the construction of lexical analyzer • Depicts the actions take place in lexical analyzer • • • states: positions in a transition diagram edges: arrows connecting the states start state: initial state of transition diagram accepting state: token recognized action: (optional) associated with a state that is executed when the state is entered start 0 > 1 = other 2 3 *

Example: Relational Operator • Transition diagram for token relop start 0 < 1 = > Return (relop, LE) 2 3 Return (relop, NE) other * 4 = > 5 6 Return (relop, LT) Return (relop, EQ) = other 7 Return (relop, GE) * 8 Return (relop, GT)

Example: id and delim id : letter or digit start 9 letter 10 other 11 * Obtain token: Either returns corresponding token or “id” return( get_token(), install_id()) Obtain attribute value: Either returns ptr to the symbol table entry or “ 0” if reserved

Example: Unsigned Numbers digit start 12 digit 13 digit . 14 digit 15 digit E 16 + or - E digit start 20 digit 21 22 digit start 25 digit 26 digit . other 27 * 17 digit 23 other 24 * 18 other 11 *

QUESTION : What would the transition diagram (TD) for strings containing each vowel, in their strict lexicographical order, look like?

Answer cons B | C | D | F | G | H | J | … | N | P | … | T | V |. . | Z string cons* A cons* E cons* I cons* O cons* U cons* cons start cons A cons E cons I cons O cons U other accept error Note: The error path is taken if the character is other than a cons or the vowel in the lex order.

Capturing Tokens Capturing keyword “begin” start b e g i Capturing variable names start A WS n WS WS – white space A – alphabetic AN – alphanumeric AN What if both need to happen at the same time?