Syntax Analyzer Parser

Syntax Analyzer Parser ASU Textbook Chapter 4.24.5, 4.7, 4.8 Tsan-sheng Hsu tshsu@iis.sinica.edu.tw http://www.iis.sinica.edu.tw/~tshsu 1

a program represented by a sequence of tokens Main tasks parser if it is a legal program, then output some abstract representation of the program Abstract representations of the input program: abstract-syntax tree + symbol table intermediate code object code Context free grammar (CFG) is used to specify the structure of legal programs. Compiler notes #3, Tsan-sheng Hsu, IIS 2

Context free grammar (CFG) Definitions: G = (T, N, P, S), where T : a set of terminals (in lower case letters); N: a set of nonterminals (in upper case letters); P : productions of the form A X 1, X 2,..., X m, where A N and X i T N; S: the starting nonterminal, S N. Notations: terminals : lower case English strings, e.g., a, b, c,... nonterminals: upper case English strings, e.g., A, B, C,... α, β, γ (T N) α, β, γ: alpha, beta and gamma. ɛ: epsilon. A X 1 A X 2 } A X 1 X 2 Compiler notes #3, Tsan-sheng Hsu, IIS 3

How does a CFG define a language? The language defined by the grammar is the set of strings (sequence of terminals) that can be derived from the starting nonterminal. How to derive something? Start with: current sequence = the starting nonterminal. Repeat find a nonterminal X in the current sequence find a production in the grammar with X on the left of the form X α, where α is ɛ or a sequence of terminals and/or nonterminals. create a new current sequence in which α replaces X Until current sequence contains no nonterminals. We derive either ɛ or a string of terminals. This is how we derive a string of the language. Compiler notes #3, Tsan-sheng Hsu, IIS 4

Example Grammar: E int E E E E E / E E ( E ) E = E E = 1 E = 1 E/E = 1 E/2 = 1 4/2 Details: The first step was done by choosing the second production. The second step was done by choosing the first production. Conventions: = : means derives in one step ; + = : means derives in one or more steps ; = : means derives in zero or more steps ; In the above example, we can write E = + 1 4/2. Compiler notes #3, Tsan-sheng Hsu, IIS 5

Language The language defined by a grammar G is L(G) = {w S + = ω}, where S is the starting nonterminal and ω is a sequence of terminals or ɛ. An element in a language is ɛ or a sequence of terminals in the set defined by the language. More terminology: E = = 1 4/2 is a derivation of 1 4/2 from E. There are several kinds of derivations that are important: The derivation is a leftmost one if the leftmost nonterminal always gets to be chosen (if we have a choice) to be replaced. It is a rightmost one if the rightmost nonterminal is replaced all the times. Compiler notes #3, Tsan-sheng Hsu, IIS 6

A way to describe derivations Construct a derivation or parse tree as follows: start with the starting nonterminal as a single-node tree REPEAT choose a leaf nonterminal X choose a production X α symbols in α become children of X UNTIL no more leaf nonterminal left Need to annotate the order of derivation on the nodes. E = E E = 1 E = 1 E/E = 1 E/2 = 1 4/2 (2) E (1) E E 1 (5) E / E 4 (3) 2 (4) Compiler notes #3, Tsan-sheng Hsu, IIS 7

Example: Grammar: E int E E E E E/E E (E) E E E Parse tree examples 1 E / E 4 2 leftmost derivation Using 1 4/2 as the input, the left parse tree is derived. A string is formed by reading the leaf nodes from left to right, which gives 1 4/2. The string 1 4/2 has another parse tree on the right. E 1 E E / E 4 E 2 rightmost derivation Some standard notations: Given a parse tree and a fixed order (for example leftmost or rightmost) we can derive the order of derivation. For the semantic of the parse tree, we normally interpret the meaning in a bottom-up fashion. That is, the one that is derived last will be serviced first. Compiler notes #3, Tsan-sheng Hsu, IIS 8

Ambiguous grammar If for grammar G and string S, there are more than one leftmost derivation for S, or more than one rightmost derivation for S, or more than one parse tree for S, then G is called ambiguous. Note: the above three conditions are equivalent in that if one is true, then all three are true. Q?: How to prove this? Hint: Any unannotated tree can be annotated with a leftmost numbering. Problems with an ambiguous grammar: Ambiguity can make parsing difficult. Underlying structure is ill-defined: in the example, the precedence is not uniquely defined, e.g., the leftmost parse tree groups 4/2 while the rightmost parse tree groups 1 4, resulting in two different semantics. Compiler notes #3, Tsan-sheng Hsu, IIS 9

Common grammar problems Lists: that is, zero or more ID s separated by commas: Note it is easy to express one or more ID s: <idlist> <idlist>, ID ID For zero or more ID s, <idlist> ɛ ID <idlist>, <idlist> won t work due to ɛ; it can generate: ID,, ID <idlist> ɛ <idlist>, ID ID won t work either because it can generate:, ID, ID We should separate out the empty list from the general list of one or more ID s. <opt-idlist> ɛ <nonemptyidlist> <nonemptyidlist> <nonemptyidlist>, ID ID Expressions: precedence and associativity as discussed next. Compiler notes #3, Tsan-sheng Hsu, IIS 10

Grammar that expresses precedence correctly Use one nonterminal for each precedence level Start with lower precedence (in our example ) Original grammar: E int E E E E E/E E (E) Revised grammar: E E E T T T/T F F int (E) E E E E T T F 1 T / T F F 4 2 rightmost derivation T / T ERROR F 2 Compiler notes #3, Tsan-sheng Hsu, IIS 11

More problems with associativity However, the above grammar is still ambiguous, and parse trees may not express the associative of and /. Example: 2 3 4 Revised grammar: E E E T T T/T F F int (E) E E E E E T T F 2 T F 3 F 4 E E E T E E rightmost derivation rightmost derivation value = (2 3) 4 = 5 value = 2 (3 4) = 3 Problems with associativity: The rule E E E has E on both sides of. Need to make the second E to some other nonterminal parsed earlier. Similarly for the rule E E/E. F 2 T F 3 T F 4 Compiler notes #3, Tsan-sheng Hsu, IIS 12

Grammar considering associative rules Original grammar: E int E E E E E/E E (E) Revised grammar: E E E T T T/T F F int (E) Final grammar: E E T T T T/F F F int (E) Example: 2 3 4 E E T E T F T F 4 F 3 2 leftmost/rightmost derivation value = (2 3) 4 = 5 Compiler notes #3, Tsan-sheng Hsu, IIS 13

Recursive productions: Rules for associativity E E T is called a left recursive production. A + = Aα. E T E is called a right recursive production. A + = αa. E E E is both left and right recursion. If one wants left associativity, use left recursion. If one wants right associativity, use right recursion. Compiler notes #3, Tsan-sheng Hsu, IIS 14

How to use CFG Breaks down the problem into pieces. Think about a C program: Declarations: typedef, struct, variables,... Procedures: type-specifier, function name, parameters, function body. function body: various statements. Example: <procedure> <type-def> ID <opt-params><opt-decl> {<opt-statements>} <opt-params> (<list-params>) <list-params> ɛ <nonemptyparlist> <nonemptyparlist> <nonemptyidlist>, ID ID One of purposes to write a grammar for a language is for others to understand. It will be nice to break things up into different levels in a top-down easily understandable fashion. Compiler notes #3, Tsan-sheng Hsu, IIS 15

Useless terms A non-terminal X is useless if either a sequence includes X cannot be derived from the starting nonterminal, or no string can be derived starting from X, where a string means ɛ or a sequence of terminals. Example 1: S A B A + ɛ B digit B digit C. B In Example 1: C is useless and so is the last production. Any nonterminal not in the right-hand side of any production is useless! Compiler notes #3, Tsan-sheng Hsu, IIS 16

More examples for useless terms Example 2: Y is useless. S X Y X ( ) Y ( Y Y ) Y derives more and more nonterminals and is useless. Any recursively defined nonterminal without a production of deriving ɛ or a string of all terminals is useless! Direct useless. Indirect useless: one can only derive direct useless terms. From now on, nonterminals. we assume a grammar contains no useless Compiler notes #3, Tsan-sheng Hsu, IIS 17

Non-context free grammars Some grammar is not CFG, that is, it may be context sensitive. Expressive power of grammars (in the order of small to large): Regular expressions FA. Context-free grammar Context-sensitive {ωcω ω is a string of a and b s} cannot be expressed by CFG. Compiler notes #3, Tsan-sheng Hsu, IIS 18

Top-down parsing There are O(n 3 )-time algorithms to parse a language defined by CFG, where n is the number of input tokens. For practical purpose, we need faster algorithms. Here we make restrictions to CFG so that we can design O(n)-time algorithms. Recursive-descent parsing : top-down parsing that allows backtracking. Attempt to find a leftmost derivation for an input string. Try out all possibilities, that is, do an exhaustive search to find a parse tree that parses the input. Compiler notes #3, Tsan-sheng Hsu, IIS 19

Example for recursive-descent parsing S cad A bc a Input: cad S S S c A d c A d c A d b c a error!! backtrack Problems with the above approach: still too slow! want to select a derivation without ever causing backtracking! trick: use lookahead symbols. Solution: use LL(1) grammars that can be parsed in O(n) time. first L: scan the input from left-to-right second L: find a leftmost derivation (1): allow one lookahead token! Compiler notes #3, Tsan-sheng Hsu, IIS 20

Predictive parser for LL(1) grammars How a predictive parser works: start by pushing the starting nonterminal into the STACK and calling the scanner to get the first token. LOOP: if top-of-stack is a nonterminal, then use the current token and the PARSING TABLE to choose a production pop the nonterminal from the STACK and push the above production s right-hand-side GOTO LOOP. if top-of-stack is a terminal and matches the current token, then pop STACK and ask scanner to provide the next token GOTO LOOP. if STACK is empty and there is no more input, then ACCEPT! If none of the above succeed, then FAIL! STACK is empty and there is input left. top-of-stack is a terminal, but does not match the current token top-of-stack is a nonterminal, but the corresponding PARSING TA- BLE entry is ERROR! Compiler notes #3, Tsan-sheng Hsu, IIS 21

Example for parsing an LL(1) grammar grammar: S ɛ (S) [S] input: ([ ]) input stack action ( S pop, push (S) ( (S) pop, match with input ([ S) pop, push [S] ([ [S]) pop, match with input ([ ] S]) pop, push ɛ ([ ] ]) pop, match with input ([ ]) ) pop, match with input ([ ]) accept S ( S ) [ S ] ε leftmost derivation Use the current input token to decide which production to derive from the top-of-stack nonterminal. Compiler notes #3, Tsan-sheng Hsu, IIS 22

About LL(1) It is not always possible to build a predictive parser given a CFG; It works only if the CFG is LL(1)! For example, the following grammar is not LL(1), but is LL(2). Grammar: S (S) [S] () [ ] Try to parse the input (). input stack action ( S pop, but use which production? In this example, we need 2-token look-ahead. If the next token is ), push (). If the next token is (, push (S). Two questions: How to tell whether a grammar G is LL(1)? How to build the PARSING TABLE? Compiler notes #3, Tsan-sheng Hsu, IIS 23

Properties of non-ll(1) grammars Theorem 1: A CFG grammar is not LL(1) if it is left-recursive. Definitions: recursive grammar: a grammar is recursive if the following is true for a nonterminal X in G: X = + αxβ. G is left-recursive if X = + Xβ. G is immediately left-recursive if X = Xβ. Compiler notes #3, Tsan-sheng Hsu, IIS 24

Example of removing immediate left-recursion Need to remove left-recursion to come out an LL(1) grammar. Example: Grammar G: A Aα β, where β does not start with A Revised grammar G : A βa A αa ɛ The above two grammars are equivalent. That is L(G) L(G ). A A Example: input baa β b α a b A a a leftmost derivation original grammar G b A a A a A ε leftmost derivation revised grammar G Compiler notes #3, Tsan-sheng Hsu, IIS 25

Rule for removing immediate left-recursion Both grammars recognize the same string, left-recursive. but G is not However, G is clear and intuitive. General rule for removing immediately left-recursion: Replace A Aα 1 Aα m β 1 β n with A β 1 A β n A A α 1 A α m A ɛ This rule does not work if α i = ɛ for some i. This is called a direct cycle in a grammar. May need to worry about whether the semantics are equivalent between the original grammar and the transformed grammar. Compiler notes #3, Tsan-sheng Hsu, IIS 26

Algorithm 4.1 Algorithm 4.1 systematically eliminates left recursion and works only if the input grammar has no cycles or ɛ-productions. Cycle: A + = A ɛ-production: A ɛ It is possible to remove cycles and all but one ɛ-production using other algorithms. Input: grammar G without cycles and ɛ-productions. Output: An equivalent grammar without left recursion. Number the nonterminals in some order A 1, A 2,..., A n for i = 1 to n do for j = 1 to i 1 do replace A i A j γ with A i δ 1 γ δ k γ where A j δ 1 δ k are all the current A j -productions. Eliminate immediate left-recursion for A i New nonterminals generated above are numbered A i+n Compiler notes #3, Tsan-sheng Hsu, IIS 27

Algorithm 4.1 Discussions Intuition: Consider only the productions where the leftmost item on the right hand side are nonterminals. If it is always the case that A i + = Aj α implies i < j, then it is not possible to have left-recursion. Why cycles are not allowed? For the procedure of removing immediate left-recursion. Why ɛ-productions are not allowed? Inside the loop, when A j ɛ, that is some δ g = ɛ, and the prefix of γ is some A k where k < i, it generates A i A k, k < i. Compiler notes #3, Tsan-sheng Hsu, IIS 28

Trace an instance of Algorithm 4.1 After each i-loop, only productions of the form A i A k γ, i < k remain. i = 1 allow A 1 A k α, k before removing immediate left-recursion remove immediate left-recursion for A 1 i = 2 j = 1: replace A 2 A 1 γ by A 2 (A k1 α 1 A kp α p )γ, where A 1 (A k1 α 1 A kp α p ) and k j > 1 k j remove immediate left-recursion for A 2 i = 3 j = 1: replace A 3 A 1 γ 1 j = 2: replace A 3 A 2 γ 2 remove immediate left-recursion for A 3 Compiler notes #3, Tsan-sheng Hsu, IIS 29

Example Original Grammar: (1) S Aa b (2) A Ac Sd e Ordering of nonterminals: S A 1 and A A 2. i = 1 do nothing as there is no immediate left-recursion for S i = 2 replace A Sd by A Aad bd hence (2) becomes A Ac Aad bd e after removing immediate left-recursion: A bda ea A ca ada ɛ resulting grammar: S Aa b A bda ea A ca ada ɛ Compiler notes #3, Tsan-sheng Hsu, IIS 30

Second property for non-ll(1) grammars Theorem 2: G is not LL(1) if a nonterminal has two productions whose right-hand-sides have a common prefix. Have left-factors. Example: S (S) () In this example, the common prefix is (. This problem can be solved by using the left-factoring trick. A αβ 1 αβ 2 Transform to: A αa A β 1 β 2 Example: S (S) () Transform to S (S S S) ) Compiler notes #3, Tsan-sheng Hsu, IIS 31

Algorithm for left-factoring Input: context free grammar G Output: equivalent left-factored context-free grammar G for each nonterminal A do find the longest non-ɛ prefix α that is common to right-hand sides of two or more productions; replace A αβ 1 αβ n γ 1 γ m with A αa γ 1 γ m A β 1 β n repeat the above process until A has no two productions with a common prefix; Compiler notes #3, Tsan-sheng Hsu, IIS 32

Left-factoring and left-recursion removal Original grammar: S (S) SS () To remove immediate left-recursion, we have S (S)S ()S S SS ɛ To do left-factoring, we have S (S S S)S )S S SS ɛ A grammar is not LL(1) if it is left recursive or has left-factors. However, grammars that are not left recursive and have no left-factors may still not be LL(1). Compiler notes #3, Tsan-sheng Hsu, IIS 33

Definition of LL(1) grammars To see if a grammar is LL(1), we need to compute its FIRST and FOLLOW sets, which are used to build its parsing table. FIRST sets: Definition: let α be a sequence of terminals and/or nonterminals or ɛ FIRST(α) is the set of terminals that begin the strings derivable from α if α can derive ɛ, then ɛ FIRST(α) FIRST(α) = {t (t is a terminal and α = tβ) or ( t = ɛ and α = ɛ)} Compiler notes #3, Tsan-sheng Hsu, IIS 34

How to compute FIRST(X)? (1/2) X is a singleton. X is a terminal: FIRST(X) = {X} X is ɛ: FIRST(X) = {ɛ} X is a nonterminal: must check all productions with X on the left-hand side. That is, for all X Y 1 Y 2 Y k perform the following steps: put FIRST(Y 1 ) {ɛ} into FIRST(X) if ɛ FIRST(Y 1 ), then put FIRST(Y 2 ) {ɛ} into FIRST(X) if ɛ FIRST(Y k 1 ), then put FIRST(Y k ) {ɛ} into FIRST(X) if ɛ FIRST(Y i ) for each 1 i k, then put ɛ into FIRST(X) Compiler notes #3, Tsan-sheng Hsu, IIS 35

How to compute FIRST(X)? (2/2) Algorithm to compute FIRST s for all non-terminals. compute FIRST s for ɛ and all terminals; initialize FIRST s for all non-terminals to ; Repeat for all nonterminals X do apply the steps to compute F IRST (X) Until no items can be added to any FIRST set; The time complexity of this algorithm. at least one item, terminal or ɛ, is added to some FIRST set in an iteration; total number of items in all FIRST sets are ( T + 1) N, where T is the set of terminals and N is the set of nonterminals. O( N 2 T ). Compiler notes #3, Tsan-sheng Hsu, IIS 36

Example for computing FIRST(X) Start with computing FIRST for the last production and walk your way up. Grammar E E T E T E ɛ T F T T / F T ɛ F int (E) H E T FIRST(F ) = {int, (} FIRST(T ) = {/, ɛ} FIRST(T ) = {int, (}, since ɛ FIRST(F ), that s all. FIRST(E ) = {, ɛ} FIRST(H) = {, int, (} FIRST(E) = {, int, (}, since ɛ FIRST(E ). FIRST(ɛ) = {ɛ} Compiler notes #3, Tsan-sheng Hsu, IIS 37

How to compute FIRST(α)? Given FIRST(X) for each terminal and nonterminal X, compute FIRST(α) for α being a sequence of terminals and/or nonterminals To build a parsing table, we need FIRST(α) for all α such that X α is a production in the grammar. Let α = X 1 X 2 X n. Perform the following steps in sequence: put FIRST(X 1 ) {ɛ} into FIRST(α) if ɛ FIRST(X 1 ), then put FIRST(X 2 ) {ɛ} into FIRST(α) if ɛ FIRST(X n 1 ), then put FIRST(X n ) {ɛ} into FIRST(α) if ɛ FIRST(X i ) for each 1 i n, then put {ɛ} into FIRST(α). Compiler notes #3, Tsan-sheng Hsu, IIS 38

Example for computing FIRST(α) Grammar E E T E T E ɛ T F T T /F T ɛ F int (E) FIRST(F ) = {int, (} FIRST(T ) = {/, ɛ} FIRST(T ) = {int, (}, since ɛ FIRST(F ), that s all. FIRST(E ) = {, ɛ} FIRST(E) = {, int, (}, since ɛ FIRST(E ). FIRST(ɛ) = {ɛ} FIRST(E T ) = {, int, (} FIRST( T E ) = { } FIRST(ɛ) = {ɛ} FIRST(F T ) = {int, (} FIRST(/F T ) = {/} FIRST(ɛ) = {ɛ} FIRST(int) = {int} FIRST((E)) = {(} FIRST(T E ) = (FIRST(T ) {ɛ}) (FIRST(E ) {ɛ}) {ɛ} Compiler notes #3, Tsan-sheng Hsu, IIS 39

Why do we need FIRST(α)? During parsing, suppose top-of-stack is a nonterminal A and there are several choices A α 1 A α 2 A α k for derivation, and the current lookahead token is a If a FIRST(α i ), then pick A α i for derivation, pop, and then push α i. If a is in several FIRST(α i ) s, then the grammar is not LL(1). Question: if a is not in any FIRST(α i ), does this mean the input stream cannot be accepted? Maybe not! What happen if ɛ is in some FIRST(α i )? Compiler notes #3, Tsan-sheng Hsu, IIS 40

FOLLOW sets Assume there is a special EOF symbol $ ends every input. Add a new terminal $. Definition: for a nonterminal X, FOLLOW(X) is the set of terminals that can appear immediately to the right of X in some partial derivation. That is, S = + α 1 Xtα 2, where t is a terminal. If X can be the rightmost symbol in a derivation, then $ is in FOLLOW(X). FOLLOW(X) = {t (t is a terminal and S = + α 1 Xtα 2 ) or ( t is $ and S = + αx)}. Compiler notes #3, Tsan-sheng Hsu, IIS 41

How to compute FOLLOW(X)? If X is the starting nonterminal, put $ into FOLLOW(X). Find the productions with X on the right-hand-side. for each production of the form Y αxβ, put FIRST(β) {ɛ} into FOLLOW(X). if ɛ FIRST(β), then put FOLLOW(Y ) into FOLLOW(X). for each production of the form Y αx, put FOLLOW(Y ) into FOLLOW(X). Repeat the above process for all nonterminals until nothing can be added to any FOLLOW set. To see if a given grammar is LL(1) and also to build its parsing table: compute FIRST(α) for every production X α compute FOLLOW(X) for all nonterminals X Note that FIRST and FOLLOW sets are always sets of terminals, plus, perhaps, ɛ for some FIRST sets. Compiler notes #3, Tsan-sheng Hsu, IIS 42

A complete example Grammar S Bc DB B ab cs D d ɛ α FIRST(α) FOLLOW(α) D {d, ɛ} {a, c} B {a, c} {c, $} S {a, c, d} {c, $} Bc {a, c} DB {d, a, c} ab {a} cs {c} d {d} ɛ {ɛ} Compiler notes #3, Tsan-sheng Hsu, IIS 43

Why do we need FOLLOW sets? Note FOLLOW(S) always includes $. Situation: During parsing, the top-of-stack is a nonterminal X and the lookahead symbol is a. Assume there are several choices for the nest derivation: X α 1 X α k If a FIRST(α gi ) for only one g i, then we use that derivation. If a FIRST(α i ) for two i, then this grammar is not LL(1). If a FIRST(α i ) for all i, then this grammar can still be LL(1)! If there exists some g i such that α gi = ɛ and a FOLLOW(X), then we can can use the derivation X α gi. Compiler notes #3, Tsan-sheng Hsu, IIS 44

Grammars that are not LL(1) A grammar is not LL(1) if there exists productions A α β and any one of the followings is true: FIRST(α) FIRST(β). ɛ FIRST(α) and FIRST(β) FOLLOW(A). ɛ FIRST(α) and ɛ FIRST(β). If a grammar is not LL(1), then you cannot write a linear-time predictive parser as described above; If a grammar is not LL(1), then we do not know to use the production A α or the production A β when the lookahead symbol is a in the following cases, respectively, a FIRST(α) FIRST(β); a FIRST(β) FOLLOW(A); a FOLLOW(A). Compiler notes #3, Tsan-sheng Hsu, IIS 45

A complete example (1/2) Grammar: <prog head> PROG ID <file list> SEMICOLON <file list> ɛ L PAREN <file list> SEMICOLON FIRST and FOLLOW sets: α FIRST(α) FOLLOW(α) ɛ {ɛ} <prog head> {PROG} {$} <file list> {ɛ, L PAREN} {SEMICOLON} PROG ID <file list> SEMICOLON {PROG} L PAREN <file list> SEMICOLON {LPAREN} Compiler notes #3, Tsan-sheng Hsu, IIS 46

A complete example (2/2) Input: PROG ID SEMICOLON Input stack action <prog head> $ PROG <prog head> $ pop, push PROG PROG ID <file list> SEMICOLON $ match input ID ID <file list> SEMICOLON $ match input SEMICOLON <file list> SEMICOLON $ WHAT TO DO? Last actions: Two choices: <file list> ɛ L PAREN <file list> SEMICOLON SEMICOLON FIRST(ɛ) and SEMICOLON FIRST(L PAREN <file list> SEMICOLON) <file list> = ɛ SEMICOLON FOLLOW(<file list>) Hence we use the derivation <file list> ɛ Compiler notes #3, Tsan-sheng Hsu, IIS 47

LL(1) parsing table (1/2) Grammar: S XC X a ɛ C a ɛ α FIRST(α) FOLLOW(α) S {a, ɛ} {$} X {a, ɛ} {a, $} C {a, ɛ} {$} ɛ {ɛ} a {a} XC {a, ɛ} Check for possible conflicts in X a ɛ. FIRST(a) FIRST(ɛ) = ɛ FIRST(ɛ) and FOLLOW(X) FIRST(a) = {a} Conflict!! ɛ FIRST(a) Check for possible conflicts in C a ɛ. FIRST(a) FIRST(ɛ) = ɛ FIRST(ɛ) and FOLLOW(C) FIRST(a) = ɛ FIRST(a) Compiler notes #3, Tsan-sheng Hsu, IIS 48

LL(1) parsing table (2/2) Parsing table: a $ S S XC S XC X conflict X ɛ C C a C ɛ Compiler notes #3, Tsan-sheng Hsu, IIS 49

Bottom-up parsing (Shift-reduce parsers) Intuition: construct the parse tree from the leaves to the root. Grammar: S AB A S B A B A Example: A x Y B w Z x w x w x w x w Y xb Z wp Input xw. This grammar is not LL(1). Compiler notes #3, Tsan-sheng Hsu, IIS 50

Rightmost derivation: S = rm S + = rm Definitions (1/2) α: the rightmost nonterminal is replaced. α: α is derived from S using one or more rightmost derivations. α is called a right-sentential form. In the previous example: S = AB = Aw = xw. rm rm rm Define similarly leftmost derivations. handle : a handle for a right-sentential form γ is the combining of the following two information: a production rule A β and a position in γ where β can be found. Compiler notes #3, Tsan-sheng Hsu, IIS 51

Definitions (2/2) Example: S aabe A Abc b B d input: abbcde γ aabcde is a right-sentential form A Abc and position 2 in γ is a handle for γ reduce : replace a handle in a right-sentential form with its left-hand-side. In the above example, replace Abc in γ with A. A right-most derivation in reverse can be obtained by handle reducing. Compiler notes #3, Tsan-sheng Hsu, IIS 52

STACK implementation Four possible actions: shift: shift the input to STACK. reduce: perform a reversed rightmost derivation. accept error STACK INPUT ACTION $ xw$ shift $x w$ reduce by A x $A w$ shift $Aw $ reduce by B w $AB $ reduce by S AB $S $ accept A S B A B A x w x w x w x w S = AB = Aw = rm rm rm xw. Compiler notes #3, Tsan-sheng Hsu, IIS 53

Viable prefix Definition: the set of prefixes of right sentential forms that can appear on the top of the stack. Some prefix cannot appear on the top of the stack during parsing. xw is a right sentential form. The prefix xw is not a viable prefix. You cannot have the situation that the suffix of xw is a handle. Note: when doing bottom-up, that is reversed rightmost derivation, the first reduction must be from a handle that is a prefix of the input string; it cannot be the case a handle on the right is reduced before a handle on the left in a right sentential form; the handle can be found on the top of the stack; Compiler notes #3, Tsan-sheng Hsu, IIS 54

Model of a shift-reduce parser stack $ s 0 s 1... sm a 0 a 1... a i... a n $ driver action table GOTO table input output Push-down automata! Current state S m encodes the symbols that has been shifted and the handles that are currently being matched. $S 0 S 1 S m a i a i+1 a n $ represents a right sentential form. GOTO table: when a reduce action is taken, which handle to replace; Action table: when a shift action is taken, which state currently in, that is, how to group symbols into handles. The power of context free grammars is equivalent to nondeterministic push down automata. Compiler notes #3, Tsan-sheng Hsu, IIS 55

LR parsers By Don Knuth at 1965. LR(k): see all of what can be derived from the right side with k input tokens lookahead. first L: scan the input from left to right second R: reverse rightmost derivation (k): with k lookahead tokens. Be able to decide the whereabout of a handle after seeing all of what have been derived so far plus k input tokens lookahead. x 1, x 2,..., x i, x i+1,..., x i+j, x i+j+1,..., x i+j+k 1,... a handle lookahead tokens Top-down parsing for LL(k) grammars: be able to choose a production by seeing only the first k symbols that will be derived from that production. Compiler notes #3, Tsan-sheng Hsu, IIS 56

LR(0) parsing Construct an FSA to recognize all possible viable prefixes. An LR(0) item ( item for short) is a production, with a dot at some position in the RHS (right-hand side). For example: A XY Z A XY Z A X Y Z A XY Z A XY Z A ɛ A The dot indicates the place of a handle. Assume G is a grammar with the starting symbol S. Augmented grammar G is to add a new starting symbol S and a new production S S to G. We assume working on the augmented grammar from now on. Compiler notes #3, Tsan-sheng Hsu, IIS 57

Closure The closure operation closure(i), where I is a set of items, is defined by the following algorithm: If A α Bβ is in closure(i), then at some point in parsing, we might see a substring derivable from Bβ as input; if B γ is a production, we also see a substring derivable from γ at this point. Thus B γ should also be in closure(i). What does closure(i) mean informally? When A α Bβ is encountered during parsing, then this means we have seen α so far, and expect to see Bβ later before reducing to A. At this point if B γ is a production, then we may also want to see B γ in order to reduce to B, and then advance to A αb β. Using closure(i) to record all possible things that we have seen in the past and expect to see in the future. Compiler notes #3, Tsan-sheng Hsu, IIS 58

Example for the closure function Example: E is the new starting symbol, and E is the original starting symbol. E E E E + T T T T F F F (E) id closure({e E}) = {E E, E E + T, E T, T T F, T F, F (E), F id} Compiler notes #3, Tsan-sheng Hsu, IIS 59

GOTO table GOT O(I, X), where I is a set of items and X is a legal symbol, means If A α Xβ is in I, then closure({a αx β}) GOT O(I, X) Informal meanings: currently we have seen A α Xβ expect to see X if we see X, then we should be in the state closure({a αx β}). Use the GOTO table to denote the state to go to once we are in I and have seen X. Compiler notes #3, Tsan-sheng Hsu, IIS 60

Sets-of-items construction Canonical LR(0) items : the set of all possible DFA states, where each state is a set of LR(0) items. Algorithm for constructing LR(0) parsing table. C {closure({s S})} repeat for each set of items I in C and each grammar symbol X such that GOT O(I, X) and not in C do add GOT O(I, X) to C until no more sets can be added to C Kernel of a state: items not of the form X β or of the form S S Given the kernel of a state, all items in the state can be derived. Compiler notes #3, Tsan-sheng Hsu, IIS 61

Example of sets of LR(0) items I 0 = closure({e E}) = {E E, Grammar: E E E E + T T T T F F F (E) id E E + T, E T, T T F, T F, F (E), Canonical LR(0) items: I 1 = GOT O(I 0, E) = {E E, E E +T } I 2 = GOT O(I 0, T ) = {E T, T T F } F id} Compiler notes #3, Tsan-sheng Hsu, IIS 62

Transition diagram (1/2) E + T * I 0 I1 I6 I9 I F I3 ( I4 id I5 T * F I2 I7 I10 ( I4 F I ( id 3 I5 ( E ) I I 4 8 I11 id T id F + I I 6 5 I2 I 3 7 Compiler notes #3, Tsan-sheng Hsu, IIS 63

Transition diagram (2/2) id I 0 E >.E E >. E+T E >.T T >.T*F T >.F F >.(E) F >.id ( F I 1 E + T E > E+. T E > E. T >. T*F E > E. + T T >.F F F >.(E) F >.id ( T I 2 E > T. T > T.*F * I 6 I 7 T > T*.F F >.(E) F >. id id I 5 I 4 F ( I 3 I 9 E > E+T. T > T.*F I 10 T > T*F. * I 7 id I 4 I 3 T > F. I 5 I 5 F > id. id I 4 F > (. E ) E >. E + T E >.T T >. T * F T >. F F >. ( E ) F >. id E F T I 8 F > ( E. ) E > E. + T I 2 ) + I 6 I 11 F > ( E ). ( I 3 Compiler notes #3, Tsan-sheng Hsu, IIS 64

Meaning of LR(0) transition diagram E + T is a viable prefix that can happen on the top of the stack while doing parsing. after seeing E + T, we are in state I 7. I 7 = {T T F, F (E), F id} We expect to follow one of the following three possible derivations: E = rm = rm = rm = rm = rm E E + T E + T F E + T id E + T F id E = rm = rm = rm = rm E E + T E + T F E + T (E) E = rm = rm = rm = rm E E + T E + T F E + T id Compiler notes #3, Tsan-sheng Hsu, IIS 65

Definitions of closure(i) and GOT O(I, X) closure(i): a state/configuration during parsing recording all possible things that we are expecting. If A α Bβ I, then it means in the middle of parsing, α is on the top of the stack; at this point, we are expecting to see Bβ; after we saw Bβ, we will reduce αbβ to A and make A top of stack. To achieve the goal of seeing Bβ, we expect to perform some operations below: We expect to see B on the top of the stack first. If B γ is a production, then it might be the case that we shall see γ on the top of the stack. If it does, we reduce γ to B. Hence we need to include B γ into closure(i). GOT O(I, X): when we are in the state described by I, and then a new symbol X is pushed into the stack, If A α Xβ is in I, then closure({a αx β}) GOT O(I, X). Compiler notes #3, Tsan-sheng Hsu, IIS 66

Parsing example Input: id * id + id STACK input action $ I 0 id*id+id$ $ I 0 id I 5 * id + id$ shift 5 $ I 0 F * id + id$ reduce by F id $ I 0 F I 3 * id + id$ in I 0, saw F, goto I 3 $ I 0 T I 2 * id + id$ reduce by T F $ I 0 T I 2 * I 7 id + id$ shift 7 $ I 0 T I 2 * I 7 id I 5 + id$ shift 5 $ I 0 T I 2 * I 7 F I 10 + id$ reduce by F id $ I 0 T I 2 + id$ reduce by T F $ I 0 E I 1 + id$ reduce by T T F $ I 0 E I 1 + I 6 id$ shift 6 $ I 0 E I 1 + I 6 id I 5 $ shift 5 $ I 0 E I 1 + I 6 F I 3 $ reduce by F id Compiler notes #3, Tsan-sheng Hsu, IIS 67

LR(0) parsing LR parsing without lookahead symbols. Constructed from DFA for recognizing viable prefixes. In state I i if A α aβ is in I i then perform shift while seeing the terminal a in the input, and then go to the state closure({a αa β}) if A β is in I i, then perform reduce by A β and then go to the state GOT O(I, A) where I is the state on the top of the stack after removing β Conflicts: shift/reduce conflict reduce/reduce conflict Very few grammars are LR(0). For example: In I 2, you can either perform a reduce or a shift when seeing * in the input However, it is not possible to have E followed by *. Thus we should not perform reduce. Use FOLLOW(E) as look ahead information to resolve some conflicts. Compiler notes #3, Tsan-sheng Hsu, IIS 68

SLR(1) parsing algorithm Using FOLLOW sets to resolve conflicts in constructing SLR(1) parsing table, where the first S stands for simple. Input: an augmented grammar G Output: the SLR(1) parsing table Construct C = {I 0, I 1,..., I n } the collection of sets of LR(0) items for G. The parsing table for state I i is determined as follows: If A α aβ is in I i and GOT O(I i, a) = I j, then action(i i, a) is shift j for a being a terminal. If A α is in I i, then action(i i, a) is reduce by A α for all terminal a FOLLOW(A); here A S If S S is in I i, then action(i i, $) is accept. If any conflicts are generated by the above algorithm, we say the grammar is not SLR(1). Compiler notes #3, Tsan-sheng Hsu, IIS 69

SLR(1) parsing table action GOTO state id + * ( ) $ E T F 0 s5 s4 1 2 3 1 s6 accept 2 r2 s7 r2 r2 3 r4 r4 r4 r4 4 s5 s4 8 2 3 5 r6 r6 r6 r6 6 s5 s4 9 3 7 s5 s4 10 8 s6 s11 9 r1 s7 r1 r1 10 r3 r3 r3 r3 11 r5 r5 r5 r5 ri means reduce by production numbered i. si means shift and then go to state I i. Use FOLLOW sets to resolve some conflicts. Compiler notes #3, Tsan-sheng Hsu, IIS 70

Discussion (1/3) Every SLR(1) grammar is unambiguous, but there are many unambiguous grammars that are not SLR(1). Example: S L = R R L R id R L States: I 0 : S S S L = R S R L R L id R L I 1 : S S I 2 : S L = R R L Compiler notes #3, Tsan-sheng Hsu, IIS 71

Discussion (2/3) I 3 : S R I 4 : L R R L L R L id I 5 : L id I 6 : S L = R R L L R L id I 7 : L R I 8 : R L id I 0 S >.S S >.L = R S >.R L >. * R L >. id R >. L * L I 5 L > id. * S I 4 L > *. R R >. L L >. * R L >. id I 1 S > S. L I 2 = R I 8 R > L. R S > L. = R R > L. I 3 S > R. I 7 L > * R. * I 6 S > L =. R R >. L L >. * R L >. id I 9 R S > L = R. I 9 : S L = R Compiler notes #3, Tsan-sheng Hsu, IIS 72

Discussion (3/3) Suppose the stack has $I 0 LI 2 and the input is =. We can either shift 6, or reduce by R L, since = FOLLOW(R). This grammar is ambiguous for SLR(1) parsing. However, we should not perform a R L reduction. After performing the reduction, the viable prefix is $R; = FOLLOW($R); = FOLLOW( R); That is to say, we cannot find a right sentential form with the prefix R =. We can find a right sentential form with R = Compiler notes #3, Tsan-sheng Hsu, IIS 73

Canonical LR LR(1) In SLR(1) parsing, if A α is in state I i, and a FOLLOW(A), then we perform the reduction A α. However, it is possible that when state I i is on the top of the stack, we have viable prefix βα on the top of the stack, and βa cannot be followed by a. In this case, we cannot perform the reduction A α. It looks difficult to find the FOLLOW sets for every viable prefix. We can solve the problem by knowing more left context using the technique of lookahead propagation. Compiler notes #3, Tsan-sheng Hsu, IIS 74

LR(1) items An LR(1) item is in the form of [A α β, a], where the first field is an LR(0) item and the second field a is a terminal belonging to a subset of FOLLOW(A). Intuition: perform a reduction based on an LR(1) item [A α, a] only when the next symbol is a. Formally: [A α β, a] is valid (or reachable) for a viable prefix γ if there exists a derivation S = δaω = rm rm δαβω, where γ = δα either a FIRST(ω) or ω = ɛ and a = $. Compiler notes #3, Tsan-sheng Hsu, IIS 75

LR(1) parsing example Grammar: S BB B ab b S = aabab = aaabab rm rm viable prefix aaa can reach [B a B, a] S = BaB = BaaB rm rm viable prefix Baa can reach [B a B, $] Compiler notes #3, Tsan-sheng Hsu, IIS 76

Finding all LR(1) items Ideas: redefine the closure function. Suppose [A α Bβ, a] is valid for a viable prefix γ δα. In other words, S = δaaω = δαbβaω. rm rm Then for each production B η, assume βaω derives the sequence of terminals bc. S = rm δαb βaω = rm δαb bc = rm δα η bc Thus [B η, b] is also valid for γ for each b FIRST(βa). Note a is a terminal. So FIRST(βa) = FIRST(βaω). Lookahead propagation. Compiler notes #3, Tsan-sheng Hsu, IIS 77

Algorithm for LR(1) parsers closure 1 (I) repeat for each item [A α Bβ, a] in I do if B η is in G then add [B η, b] to I for each b FIRST(βa) until no more items can be added to I return I GOT O 1 (I, X) let J = {[A αx β, a] [A α Xβ, a] I}; return closure 1 (J) items(g ) C {closure 1 ({[S S, $]})} repeat for each set of items I C and each grammar symbol X such that GOT O 1 (I, X) and GOT O 1 (I, X) C do add GOT O 1 (I, X) to C until no more sets of items can be added to C Compiler notes #3, Tsan-sheng Hsu, IIS 78

Example for constructing LR(1) closures Grammar: S S S CC C cc d closure 1 ({[S S, $]}) = {[S S, $], [S CC, $], [C cc, c/d], [C d, c/d]} Note: FIRST(ɛ$) = {$} FIRST(C$) = {c, d} [C cc, c/d] means [C cc, c] and [C cc, d]. Compiler notes #3, Tsan-sheng Hsu, IIS 79

LR(1) Transition diagram d I 0 S >. S, $ S >. CC, $ C >. cc, c/d C >.d, c/d d c I 3 C C > c.c, c/d C >.cc, c/d C >.d, c/d S I 2 S > C.C, $ C >.cc, $ C >.d, $ C c I 8 C > cc., c/d I 1 S > S., $ C c d I 5 S > CC., $ I 6 C > c.c, $ C >.cc, $ C >.d, $ d I 7 C > d., $ c C I 9 C > cc., $ I 4 C > d., c/d Compiler notes #3, Tsan-sheng Hsu, IIS 80

LR(1) parsing example Input cdccd STACK INPUT ACTION $ I 0 cdccd$ $ I 0 c I 3 dccd$ shift 3 $ I 0 c I 3 d I 4 ccd$ shift 4 $ I 0 c I 3 C I 8 ccd$ reduce by C d $ I 0 C I 2 ccd$ reduce by C cc $ I 0 C I 2 c I 6 cd$ shift 6 $ I 0 C I 2 c I 6 c I 6 d$ shift 6 $ I 0 C I 2 c I 6 c I 6 d$ shift 6 $ I 0 C I 2 c I 6 c I 6 d I 7 $ shift 7 $ I 0 C I 2 c I 6 c I 6 C I 9 $ reduce by C cc $ I 0 C I 2 c I 6 C I 9 $ reduce by C cc $ I 0 C I 2 C I 5 $ reduce by S CC $ I 0 S I 1 $ reduce by S S $I 0 S $ accept Compiler notes #3, Tsan-sheng Hsu, IIS 81

Generating LR(1) parsing table Construction of canonical LR(1) parsing tables. Input: an augmented grammar G Output: the canonical LR(1) parsing table, i.e., the ACT ION 1 table Construct C = {I 0, I 1,..., I n } the collection of sets of LR(1) items form G. Action table is constructed as follows: if [A α aβ, b] I i and GOT O 1 (I i, a) = I j, then action 1 [I i, a] = shift j for a is a terminal. if [A α, a] I i and A S, then action 1 [I i, a] = reduce by A α if [S S., $] I i, then action 1 [I i, $] = accept. If conflicts result from the above rules, then the grammar is not LR(1). The initial state of the parser is the one constructed from the set containing the item [S S, $]. Compiler notes #3, Tsan-sheng Hsu, IIS 82

Example of an LR(1) parsing table action 1 GOTO 1 state c d $ S C 0 s3 s4 1 2 1 accept 2 s6 s7 5 3 s3 s4 8 4 r3 r3 5 r1 6 s6 s7 9 7 r3 8 r2 r2 9 r2 Canonical LR(1) parser: Most powerful! Has too many states and thus occupy too much space. Compiler notes #3, Tsan-sheng Hsu, IIS 83

LALR(1) parser Lookahead LR The method that is often used in practice. Most common syntactic constructs of programming languages can be expressed conveniently by an LALR(1) grammar. SLR(1) and LALR(1) always have the same number of states. Number of states is about 1/10 of that of LR(1). Simple observation: an LR(1) item is of the form [A α β, c] We call A α β the first component. Definition: in an LR(1) state, set of first components is called its core. Compiler notes #3, Tsan-sheng Hsu, IIS 84

Intuition for LALR(1) grammars In an LR(1) parser, it is a common thing that several states only differ in lookahead symbols, but have the same core. To reduce the number of states, we might want to merge states with the same core. If I 4 and I 7 are merged, then the new state is called I 4,7 After merging the states, revise the GOT O 1 table accordingly. Merging of states can never produce a shift-reduce conflict that was not present in one of the original states. I 1 = {[A α, a],...} I 2 = {[B β aγ, b],...} For I 1, we perform a reduce on a. For I 2, we perform a shift on a. Merging I 1 and I 2, the new state I 1,2 has shift-reduce conflicts. This is impossible! In the original table, I 1 and I 2 have the same core. [A α, c] I 2 and [B β aγ, d] I 1. The shift-reduce conflict already occurs in I 1 and I 2. Compiler notes #3, Tsan-sheng Hsu, IIS 85

LALR(1) Transition diagram d I 0 S >. S, $ S >. CC, $ C >. cc, c/d C >.d, c/d c I 3 C C > c.c, c/d C >.cc, c/d C >.d, c/d S I 2 S > C.C, $ C >.cc, $ C >.d, $ c I 1 S > S., $ C c d I 5 S > CC., $ I 6 C > c.c, $ C >.cc, $ C >.d, $ c C d C d I 8 C > cc., c/d I 9 C > cc., $ I 4 C > d., c/d I 7 C > d., $ Compiler notes #3, Tsan-sheng Hsu, IIS 86

Possible new conflicts from LALR(1) May produce a new reduce-reduce conflict. For example (textbook page 238), grammar: S S S aad bbf abe bae A c B c The language recognized by this grammar is {acd, ace, bcd, bce}. You may check that this grammar is LR(1) by constructing the sets of items. You will find the set of items {[A c, d], [B c, e]} is valid for the viable prefix ac, and {[A c, e], [B c, d]} is valid for the viable prefix bc. Neither of these sets generates a conflict, and their cores are the same. However, their union, which is {[A c, d/e], [B c, d/e]} generates a reduce-reduce conflict, since reductions by both A c and B c are called for on inputs d and e. Compiler notes #3, Tsan-sheng Hsu, IIS 87

How to construct LALR(1) parsing table Naive approach: Construct LR(1) parsing table, which takes lots of intermediate spaces. Merging states. Space efficient methods to construct an LALR(1) parsing table are known. Construction and merging on the fly. Compiler notes #3, Tsan-sheng Hsu, IIS 88

Summary LALR(1) LALR(1) LL(1) LR(1) SLR(1) LR(1) SLR(1) LR(0) LR(1) and LALR(1) can almost handle all programming languages, but LALR(1) is easier to write and uses much less space. LL(1) is easier to understand, but cannot handle several important common-language features. Compiler notes #3, Tsan-sheng Hsu, IIS 89