CS 321 Programming Languages and Compilers VI Parsing

CS 321 Programming Languages and Compilers VI. Parsing

Parsing Calculate grammatical structure of program, like diagramming sentences, where: Tokens = “words” Programs = “sentences” For further information, read: Aho, Sethi, Ullman, “Compilers: Principles, Techniques, and Tools” (a. k. a, the “Dragon Book”) 2 Parsing

Outline of coverage • Context-free grammars • Parsing – Tabular Parsing Methods – One pass » Top-down » Bottom-up • Yacc 3 Parsing

What parser does: Extracts grammatical structure of program function-def name arguments stmt-list stmt main expression operator expression variable << string cout “hello, worldn” 4 Parsing

Context-free languages Grammatical structure defined by context-free grammar. statement labeled-statement | expression-statement | compound-statement labeled-statement ident : statement | case constant-expression : statement compound-statement { declaration-list statement-list } “Context-free” = only one non-terminal in left-part. terminal non-terminal 5 Parsing

Parse trees Parse tree = tree labeled with grammar symbols, such that: • If node is labeled A, and its children are labeled x 1. . . xn, then there is a production A x 1. . . xn • “Parse tree from A” = root labeled with A • “Complete parse tree” = all leaves labeled with tokens 6 Parsing

Parse trees and sentences Frontier of tree = labels on leaves (in left-to-right order) Frontier of tree from S is a sentential form. Frontier of a complete tree from S is a sentence. L L ; E E “Frontier” a 7 Parsing

Example G: L L ; E | E E a | b Syntax trees from start symbol (L): L E a L L L ; E L E E a L ; ; E b a Sentential forms: a a; E a; b; b 8 Parsing

Derivations Alternate definition of sentence: • Given , in V*, say is a derivation step if ’ ’’ and = ’ ’’ , where A is a production • is a sentential form iff there exists a derivation (sequence of derivation steps) S ( alternatively, we say that S * ) Two definitions are equivalent, but note that there are many derivations corresponding to each parse tree. 9 Parsing

Another example H: L E ; L | E E a | b L L E ; L E a E L E ; b L E b a ; L E a 10 Parsing

Ambiguity • For some purposes, it is important to know whether a sentence can have more than one parse tree. • A grammar is ambiguous if there is a sentence with more than one parse tree. • Example: E E+E | E*E | id E E E + E id E * id E E E id id 11 + * E E id id Parsing

Ambiguity • Ambiguity is a function of the grammar rather than the language. Certain unambiguous grammars may have equivalent ambiguous ones. 12 Parsing

Grammar Transformations • Grammars can be transformed without affecting the language generated. • Three transformations are discussed next: – Eliminating Ambiguity – Eliminating Left Recursion (i. e. productions of the form A A ) – Left Factoring 13 Parsing

Grammar Transformation 1. Eliminating Ambiguity • Sometimes an ambiguous grammar can be rewritten to eliminate ambiguity. • For example, expressions involving additions and products can be written as follows: E E+T | T T T*id | id • The language generated by this grammar is the same as that generated by the grammar on tranparency 11. Both generate id(+id|*id)* • However, this grammar is not ambiguous. 14 Parsing

Grammar Transformation 1. Eliminating Ambiguity (Cont. ) • One advantage of this grammar is that it represents the precedence between operators. In the parsing tree, products appear nested within additions E E T id + T T * id id 15 Parsing

Grammar Transformation 1. Eliminating Ambiguity (Cont. ) • The most famous example of ambiguity in a programming language is the dangling else. • Consider S if then S else S | if then S | 16 Parsing

Grammar Transformation 1. Eliminating Ambiguity (Cont. ) • When there are two nested ifs and only one else. . S if then else S S if then S S if then if S then S 17 else S Parsing

Grammar Transformation 1. Eliminating Ambiguity (Cont. ) • In most languages (including C++ and Java), each else is assumed to belong to the nearest if that is not already matched by an else. This association is expressed in the following (unambiguous) grammar: S Matched | Unmatched Matched if then Matched else Matched Unmatched if then S | if then Matched else Unmatched | 18 Parsing

Grammar Transformation 1. Eliminating Ambiguity (Cont. ) • Ambiguity is a function of the grammar • It is undecidable whether a context free grammar is ambiguous. • The proof is done by reduction to Post’s correspondence problem. • Although there is no general algorithm, it is possible to isolate certain constructs in productions which lead to ambiguous grammars. 19 Parsing

Grammar Transformation 1. Eliminating Ambiguity (Cont. ) • For example, a grammar containg the production A AA | would be ambiguous, because the substring has two parses. A A A A A A A • This ambiguity disappears if we use the productions A AB | B and B • or the productions A BA | B and B . 20 Parsing

Grammar Transformation 1. Eliminating Ambiguity (Cont. ) • Other three examples of ambiguous productions are: – A A A – A A | A and – A A | A A • A language generated by an ambiguous Context Free Grammar is inherently ambiguous if it has no unambiguous Context Free Grammar. (This can be proven formally) – An example of such a language is L={aibjcm | i=j or j=m} which can be generated by the grammar: S AB | DC A a. A | C c. C | B b. Bc | D a. Db | 21 Parsing

Grammar Transformations 2. Elimination of Left Recursion • A grammar is left recursive if it has a nonterminal A and a derivation A +A for some string Top-down parsing methods (to be discussed shortly) cannot handle left-recursive grammars, so a transformation to eliminate left recursion is needed. • Immediate left recursion (productions of the form A A ) can be easily eliminated. • We group the A-productions as A A 1 | A 2 | … | A m | 1| 2 | … | n where no i begins with A. Then we replace the Aproductions by A 1 A’ | 2 A’ | … | n A’ A’ 1 A’ | 2 A’| … | m A’ | 22 Parsing

Grammar Transformations 2. Elimination of Left Recursion (Cont. ) • The previous transformation, however, does not eliminate left recursion involving two or more steps. For example, consider the grammar S Aa | b A Ac| Sd |e S is left-recursive because S Aa Sda , but it is not immediately left recursive. 23 Parsing

Grammar Transformations 2. Elimination of Left Recursion (Cont. ) 1. Algorithm. Eliminate left recursion 2. Arrange nonterminals in some order A 1, A 2 , , …, An 3. for i =1 to n { 4. for j =1 to i -1 { 5. 6. g by the production Ai d 1 g | d 2 g | … | dn g where Aj d 1 | d 2 |…| dn are all the current Aj-productions replace each production of the form Ai Aj 7. 8. } 9. eliminate the immediate left recursion among the Ai-productions 10. } 24 Parsing

Grammar Transformations 2. Elimination of Left Recursion (Cont. ) • To show that the previous algorithm actually works all we need notice is that iteration i only changes productions with Ai on the left-hand side. And m > i in all productions of the form Ai Am . • This can be easily shown by induction. – It is clearly true for i=1. – If it is true for all i<k, then when the outer loop is executed for i=k, the inner loop will remove all productions Ai Am with m < i. – Finally, with the elimination of self recursion, m in the Ai Am productions is forced to be > i. • So, at the end of the algorithm, all derivations of the form Ai +Am will have m > i and therefore left recursion would not be possible. 25 Parsing

Grammar Transformations 3. Left Factoring • Left factoring helps transform a grammar for predictive parsing • For example, if we have the two productions S if then S else S | if then S • on seeing the input token if, we cannot immediately tell which production to choose to expand S. • In general, if we have A 1 | 2 and the input begins with , we do not know (without looking further) which production to use to expand A. 26 Parsing

Grammar Transformations 3. Left Factoring(Cont. ) • However, we may defer the decision by expanding A to A’. • Then after seeing the input derived from , we may expand A’ to 1 or to 2. That is, left-factored, the original productions become A A’ A’ 1 | 2 27 Parsing

Non-Context-Free Language Constructs • Examples of non-context-free languages are: – L 1={wcw | w is of the form (a|b)*} – L 2={anbmcndm | n 1 and m 1 } – L 3={anbncn | n 0 } • Languages similar to these that are context free – L’ 1={wcw. R | w is of the form (a|b)*} (w. R stands for w reversed) This language is generated by the grammar » S a. Sa | b. Sb | c – L’ 2={anbmcmdn | n 1 and m 1 } This language is generated by the grammar » S a. Sd | a. Ad » A b. Ac | bc 28 Parsing

Non-Context-Free Language Constructs (Cont. – L” 2={anbncmdm | n 1 and m 1 } This language is generated by the grammar » S AB » A a. Ab | ab » B c. Bd | cd – L’ 3={anbn | n 1} This language is generated by the grammar » S a. Sb | ab This language is not definable by any regular expression 29 Parsing

Non-Context-Free Language Constructs (Cont. • Suppose we could construct a DFSM D accepting L’ 3. • D must have a finite number of states, say k. • Consider the sequence of states s 0, s 1, s 2, …, sk entered by D having read , a, aa, …, ak. • Since D only has k states, two of the states in the sequence have to be equal. Say, si sj (i j). • From si, a sequence of i bs leads to an accepting (final) state. Therefore, the same sequence of i bs will also lead to an accepting state from sj. Therefore D would accept ajbi which means that the language accepted by D is not identical to L’ 3. A contradiction. 30 Parsing

Parsing The parsing problem is: Given string of tokens w, find a parse tree whose frontier is w. (Equivalently, find a derivation from w. ) A parser for a grammar G reads a list of tokens and finds a parse tree if they form a sentence (or reports an error otherwise) Two classes of algorithms for parsing: – Top-down – Bottom-up 31 Parsing

Parser generators A parser generator is a program that reads a grammar and produces a parser. The best known parser generator is yacc. Both produce bottom-up parsers. Most parser generators - including yacc - do not work for every cfg; they accept a restricted class of cfg’s that can be parsed efficiently using the method employed by that parser generator. 32 Parsing

Top-down parsing • Starting from parse tree containing just S, build tree down toward input. Expand left-most nonterminal. • Algorithm: (next slide) 33 Parsing

Top-down parsing (cont. ) • Let input = a 1 a 2. . . an current sentential form (csf) = S loop { suppose csf = t 1. . . tk. A if t 1. . . tk a 1. . . ak , it’s an error based on ak+1. . . , choose production A csf becomes t 1. . . tk } 34 Parsing

Top-down parsing example Grammar: H: L E ; L | E E a | b Input: a; b Parse tree Sentential form L L E ; L Input L E; L a; b a; L a; b a 35 Parsing

Top-down parsing example (cont. ) Parse tree L E ; L a Sentential form a; E a; b E L E ; L a Input E b 36 Parsing

LL(1) parsing Efficient form of top-down parsing. Use only first symbol of remaining input (ak+1) to choose next production. That is, employ a function M: N P in “choose production” step of algorithm. When this works, grammar is (usually) called LL(1). (More precise definition to follow. ) 37 Parsing

LL(1) examples • Example 1: H: L E ; L | E E a | b Given input a; b, so next symbol is a. Which production to use? Can’t tell. H not LL(1). 38 Parsing

LL(1) examples • Example 2: Exp Term Exp’ $ | + Exp Term id (Use $ for “end-of-input” symbol. ) Grammar is LL(1): Exp and Term have only one production; Exp’ has two productions but only one is applicable at any time. 39 Parsing

Nonrecursive predictive parsing • It is possible to build a nonrecursive predictive parser by maintaining as stack explicitly, rather tan implicitly via recursive calls. • The key problem during predictive parsing is that of determining the production to be applied for a non-terminal. 40 Parsing

Nonrecursive predictive parsing Algorithm. Nonrecursive predictive parsing Set ip to point to the first symbol of w$. repeat Let X be the top of the stack symbol and a the symbol pointed to by ip if X is a terminal or $ then if X == a then pop X from the stack and advance ip else error() else // X is a nonterminal if M[X, a] == X Y 1 Y 2 … Y k then pop X from the stack push Yk. Y k-1, …, Y 1 onto the stack with Y 1 on top (push nothing if Y 1 Y 2 … Y k is ) output the production X Y 1 Y 2 … Y k else error() until X == $ 41 Parsing

LL(1) grammars • No left recursion. A A : If this production is chosen, parse makes no progress. • No common prefixes. A | Can fix by “left factoring”: A A’ ’ | 42 Parsing

LL(1) grammars (cont. ) • No ambiguity. Precise definition requires that production to choose be unique (“choose” function M very hard to calculate otherwise). 43 Parsing

Top-down Parsing L Input tokens: <t 0, t 1, …, t-i, . . . > Start symbol and root of parse tree E 0 … E-n L Input tokens: <t-i, . . . > E 0 … E-n From left to right, “grow” the parse tree downwards . . . 44 Parsing

Checking LL(1)-ness • For any sequence of grammar symbols , define set FIRST( ) to be those tokens a such that … a for some . (Notation: write *a. ) 45 Parsing

Checking LL(1)-ness • Define: Grammar G = (N, , P, S) is LL(1) if whenever there are two left-most derivations (in which the leftmost non-terminal is always expanded first ) S =>* w. A => w =>* wx S =>* w. A => w =>* wy Such that FIRST(x) = FIRST(y), it follows that =. In other words, given 1. A string w. A in V* and 2. The first terminal symbol to be derived from A , say t There is at most one production that can be applied to A to yield a derivation of any terminal string beginning with wt. • FIRST sets can often be calculated by inspection. 46 Parsing

FIRST Sets Exp Term Exp’ $ | + Exp Term id (Use $ for “end-of-input” symbol. ) • FIRST(Term Exp’) = {id} • FIRST($) = {$}, FIRST(+ Exp) = {+} implies FIRST($) FIRST(+ Exp) = {} • FIRST(id) = {id} grammar is LL(1) 47 Parsing

FIRST Sets H: L E ; L | E E a | b FIRST(E ; L) = {a, b} = FIRST(E) FIRST(E ; L) FIRST(E) {} H not LL(1). 48 Parsing

How to compute FIRST Sets of Vocabulary Sym Algorithm. Compute FIRST(X) for all grammar symbols X forall X V do FIRST(X)={} forall X (X is a terminal) do FIRST(X)={X} forall productions X do FIRST(X) = FIRST(X) U { } repeat forall productions X Y 1 Y 2 … Y k do forall i [1, k] do FIRST(X) = FIRST(X) U (FIRST(Yi) - { }) if FIRST(Y i ) then continue outer loop FIRST(X) = FIRST(X) U { } until no more terminals or are added to any FIRST set 49 Parsing

How to compute FIRST Sets of Strings of Symb • FIRST(X 1 X 2…Xn) is the union of FIRST(X 1) and all FIRST(Xi) such that FIRST(X k ) for k=1, 2, . . , i-1 • FIRST(X 1 X 2…Xn) contains iff FIRST(X k ) for k=1, 2, . . , n. 50 Parsing

FIRST Sets do not Suffice • Given the productions Tx A T y T w T • T w should be applied when the next input token is w. • T should be applied whenever the next terminal (the one pointed to by ip) is either x or y A 51 Parsing

FOLLOW Sets • For any nonterminal X, define set FOLLOW(X) to be those tokens a such that S * Xa for some and . 52 Parsing

How to compute the FOLLOW Set Algorithm. Compute FOLLOW(X) for all nonterminals X FOLLOW(S) ={$} forall productions A B do FOLLOW(B)=Follow(B) U (FIRST( ) - { }) repeat forall productions A B or A B with FIRST( ) do FOLLOW(B) = FOLLOW(B) U FOLLOW(A) until all FOLLOW sets remain the same 53 Parsing

Construction of a predictive parsing table Algorithm. Construction of a predictive parsing table M[: , : ] = {} forall productions A do forall a FIRST( ) do M[A, a] = M[A, a] U {A } if FIRST( ) then forall b FOLLOW(A) do M[A, b] = M[A, b] U {A } Make all empty entries of M be error 54 Parsing

Another Definition of LL(1) Define: Grammar G is LL(1) if for every A N with productions A 1 | | n, FIRST( i FOLLOW(A)) FIRST( j FOLLOW(A) ) = for all i, j; 55 Parsing