Lexical and Syntactic Analysis Here we look at

  • Slides: 12
Download presentation
Lexical and Syntactic Analysis • Here, we look at two of the tasks involved

Lexical and Syntactic Analysis • Here, we look at two of the tasks involved in the compilation process – Given source code, we need to first break it into meaningful units (lexical analysis) and then parse these lexical components into their syntactic uses (syntactic analysis) • The goal of lexical analysis is to identify each lexeme in the source code and assign it the proper token • The goal of syntactic analysis is to parse the lexemes into a parse tree – During both lexical and syntactic analysis, errors can be detected and reported – Note that this two-step process omits the actual translation from source code to machine language – this is also required, but we will not consider it here

Lexical Analysis • Given source code that consists of – reserved words, identifiers, punctuation,

Lexical Analysis • Given source code that consists of – reserved words, identifiers, punctuation, blank spaces, comments • Identify each item for what it is – if it is a reserved word or punctuation, categorize the type, if it is an identifier, add it to the symbol table (if not already present), if it is blank space or comment, discard (ignore) it • How will we perform this operation? – we can use a relatively simple state transition diagram to describe the various entities of interest and then implement this in a program • the lexical analysis program’s task is to parse the input • and produce each item individually (the lexeme) • each item should include its type (the token)

Recognizing Names/Words/Numbers int lex( ) { get. Char( ); switch (char. Class) { case

Recognizing Names/Words/Numbers int lex( ) { get. Char( ); switch (char. Class) { case LETTER: add. Char( ); get. Char( ); while (char. Class == LETTER || char. Class == DIGIT) { add. Char( ); get. Char( ); } return lookup(lexeme); break; case DIGIT: add. Char( ); get. Char( ); while (char. Class == DIGIT) { add. Char( ); get. Char( ); } return INT_LIT; break; } /* End of switch */ } /* End of function lex */

 • The process of Parsing – generating a parse tree from a set

• The process of Parsing – generating a parse tree from a set of input that identifies • the grammatical categories of each element of the input – identifying if and where errors occur • Parsing is similar whether for a natural language or a programming language – a good parser will continue parsing even after errors have been found • this requires a recovery process • Two general forms of parsers – Top-down (used in LL parser algorithm) • start with LHS rules, map to RHS rules until terminal symbols have been identified, match these against the input – Bottom-up (used in LR parser algorithms) • start with RHS rules and input, collapse terminals and non-terminals into nonterminals until you have reached the starting non-terminal – Parsing is an O(n 3) problem where n is the number of items in the input • if we cannot determine a single token for each lexeme, the problem because O(2 n)! • by restricting our parser to work only on the grammar of the given language, we can reduce the complexity to O(n)

Top-Down Parsing • Using a BNF of a language, we generate a recursive-decent parser

Top-Down Parsing • Using a BNF of a language, we generate a recursive-decent parser – each of our non-terminal grammatical categories in the BNF are converted into functions (e. g. , <expr>, <if>, <factor>, etc) – in any given function, when called, it parses the next lexeme using a function called lex( ), and maps it to terminal symbols and/or calls further functions • this approach is known as an LL Parser – left-to-right parse, using leftmost derivations – it is simple to generate the recursive-decent parser • There are two restrictions that we must make on the grammar – the grammar specifying the language cannot have left recursion • if a rule has recursive parts, those parts must not be the first items on the RHS of a rule – the grammars must pass the pairwise disjointness test • Algorithms exist to alter a grammar so that it passes both restrictions

Recursive Decent Parser Example Recall our example expression grammar from chapter 3: <expr> <term>

Recursive Decent Parser Example Recall our example expression grammar from chapter 3: <expr> <term> {(+ | -) <term>} <term> <factor> {(* | /) <factor>} <factor> id | ( <expr> ) void expr( ) { term( ); while (next. Token = = PLUS_CODE || next. Token = = MINUS_CODE){ lex( ); term( ); } } void term( ) { factor( ); while (next. Token = = MULT_CODE || next. Token = = DIV_CODE) { lex( ); factor( ); } } void factor( ) { if(next. Token = = ID_CODE) lex( ); else if(next. Token = = LEFT_PAREN_CODE) { lex( ); expr( ); if(next. Token = = RIGHT_PAREN_CODE) lex( ); else error( ); } else error( ); }

 void ifstmt( ) { if (next. Token != IF_CODE) error( ); else {

void ifstmt( ) { if (next. Token != IF_CODE) error( ); else { lex( ); if (next. Token != LEFT_PAREN_CODE) error( ); else { boolexpr( ); if (next. Token != RIGHT_PAREN_CODE) error( ); else { statement( ); if(next. Token = = ELSE_CODE) { lex( ); statement( ); } } } If Statement Example We expect an if statement to look like this: if (boolean expr) statement; optionally followed by: else statement; Otherwise, we return an error

LL Parser Restriction • Recall one of our restrictions for the use of the

LL Parser Restriction • Recall one of our restrictions for the use of the LL parser was that the grammar pass the pairwise disjointness test – The parser will need to be able to select the proper right-hand side rule to apply while parsing • if the current rule being applied is of a <factor>, should we apply <id> or (<expr>) to it? – For the parser to be able to know which rule to apply, the first non-terminal on each right-hand side rule must differ • consider a rule – <A> a<B> | a<C> • if the parser finds an “a” in the input, which rule should be applied – should it call function B or C?

Pairwise Disjointness Test • The pairwise disjointness test examines a grammar to make sure

Pairwise Disjointness Test • The pairwise disjointness test examines a grammar to make sure that – all rules in the grammar are pairwise disjoint for the same LHS • the book provides a formal definition that we will skip – here are some examples • A a. B | b. Ab | c – is pairwise disjoint • A a. B | a. Ab – is not pairwise disjoint • <var> id | id[<expr>] – is not pairwise disjoint, but we can make it so: • <var> id<next> • <next> e | [<expr>] – e means empty set

Bottom-Up Parsing • Because of the two restrictions placed on grammars to qualify for

Bottom-Up Parsing • Because of the two restrictions placed on grammars to qualify for the LL parser – an alternative approach is the LR parser which does bottom-up parsing • LR: Left-to-right parsing, Rightmost derivation • The parser is implemented using a pushdown automaton – a stack added to the state diagrams seen earlier • The parser has two basic processes – shift – move items from the input onto the stack – reduce – take consecutive stack items and reduce them, for instance, if we have a rule <A> a<B> and we have a and <B> on the stack, reduce them to <A> • while the parser is easy to implement, it relies on an LR parsing table, which is difficult to generate – there are numerous algorithms to generate the parsing table, we will skip how to do that and assume we already have one available

Parser Algorithm • Given input S 0, a 1, …, an, $ – S

Parser Algorithm • Given input S 0, a 1, …, an, $ – S 0 is the start state – a 1, …, an are the lexemes that make up the program – $ is a special end of input symbol • If action[Sm, ai] = Shift S, then push ai, S onto stack and change state to S • If action[Sm, ai] = Reduce R, then use rule R in the grammar and reduce the items on the stack appropriately, changing state to be the state GOTO[Sm, R] • If action[Sm, ai] = Accept then the parse is complete with no errors The Parsing • If action[Sm, ai] = table stores the Error (or the entry values of in the table is action[x, y] and blank) then call GOTO[x, y] error-handling and recovery routine

Example Parse of id+id*id$ Stack Input Action 0 id+id*id$ S 5 0 id 5

Example Parse of id+id*id$ Stack Input Action 0 id+id*id$ S 5 0 id 5 +id*id$ R 6(GOTO[0, F]) 0 F 3 +id*id$ R 4(GOTO[0, T]) 0 T 2 +id*id$ R 2(GOTO[0, E]) 0 E 1 +id*id$ S 6 0 E 1+6 id*id$ S 5 0 E 1+6 id 5 *id$ R 6(GOTO[6, F]) 0 E 1+6 F 3 *id$ R 4(GOTO[6, T]) 0 E 1+6 T 9 *id$ S 7 0 E 1+6 T 9*7 id$ S 5 0 E 1+6 T 9*7 id 5 $ R 6(GOTO[7, F]) 0 E 1+6 T 9*7 F 10 $ R 3(GOTO[6, T]) 0 E 1+6 T 9 $ R 1(GOTO[0, E]) 0 E 1 $ ACCEPT Grammar: 1. E E + T 2. E T 3. T T * F 4. T F 5. F (E) 6. F id