LINGUIST 180 Introduction to Computational Linguistics Dan Jurafsky

  • Slides: 73
Download presentation
LINGUIST 180: Introduction to Computational Linguistics Dan Jurafsky, Marie-Catherine de Marneffe Lecture 9: Grammar

LINGUIST 180: Introduction to Computational Linguistics Dan Jurafsky, Marie-Catherine de Marneffe Lecture 9: Grammar and Parsing (I) Thanks to Jim Martin for many of these slides! LING 180 Autumn 2007 1

Outline for Grammar/Parsing Week Context-Free Grammars and Constituency Some common CFG phenomena for English

Outline for Grammar/Parsing Week Context-Free Grammars and Constituency Some common CFG phenomena for English Sentence-level constructions NP, PP, VP Coordination Subcategorization Top-down and Bottom-up Parsing Dynamic Programming Parsing Quick sketch of probabilistic parsing LING 180 Autumn 2007 2

Review Parts of Speech Basic syntactic/morphological categories that words belong to Part of Speech

Review Parts of Speech Basic syntactic/morphological categories that words belong to Part of Speech tagging Assigning parts of speech to all the words in a sentence LING 180 Autumn 2007 3

Syntax: from Greek syntaxis “setting out together, arrangement’’ Refers to the way words are

Syntax: from Greek syntaxis “setting out together, arrangement’’ Refers to the way words are arranged together, and the relationship between them. Distinction: Prescriptive grammar: how people ought to talk Descriptive grammar: how they do talk Goal of syntax is to model the knowledge of that people unconsciously have about the grammar of their native language LING 180 Autumn 2007 4

Syntax Why should we care? Grammar checkers Question answering Information extraction Machine translation LING

Syntax Why should we care? Grammar checkers Question answering Information extraction Machine translation LING 180 Autumn 2007 5

key ideas of syntax Constituency (we’ll spend most of our time on this) Subcategorization

key ideas of syntax Constituency (we’ll spend most of our time on this) Subcategorization Grammatical relations Plus one part we won’t have time for: Movement/long-distance dependency LING 180 Autumn 2007 6

Context-Free Grammars (CFG) Capture constituency and ordering Ordering: – What are the rules that

Context-Free Grammars (CFG) Capture constituency and ordering Ordering: – What are the rules that govern the ordering of words and bigger units in the language? Constituency: How words group into units and how the various kinds of units behave LING 180 Autumn 2007 7

Constituency E. g. , Noun phrases (NPs) – – – Three parties from Brooklyn

Constituency E. g. , Noun phrases (NPs) – – – Three parties from Brooklyn A high-class spot such as Mindy’s The Broadway coppers They Harry the Horse The reason he comes into the Hot Box How do we know these form a constituent? LING 180 Autumn 2007 8

Constituency (II) They can all appear before a verb: – – Three parties from

Constituency (II) They can all appear before a verb: – – Three parties from Brooklyn arrive… A high-class spot such as Mindy’s attracts… The Broadway coppers love… They sit But individual words can’t always appear before verbs: – – *from arrive… *as attracts… *the is *spot is… Must be able to state generalizations like: – Noun phrases occur before verbs LING 180 Autumn 2007 9

Constituency (III) Preposing and postposing: On September 17 th, I’d like to fly from

Constituency (III) Preposing and postposing: On September 17 th, I’d like to fly from Atlanta to Denver I’d like to fly on September 17 th from Atlanta to Denver I’d like to fly from Atlanta to Denver on September 17 th. But not: *On September, I’d like to fly 17 th from Atlanta to Denver *On I’d like to fly September 17 th from Atlanta to Denver LING 180 Autumn 2007 10

CFG example S -> NP VP NP -> Det NOMINAL -> Noun VP ->

CFG example S -> NP VP NP -> Det NOMINAL -> Noun VP -> Verb Det -> a Noun -> flight Verb -> left LING 180 Autumn 2007 11

CFGs: set of rules S -> NP VP This says that there are units

CFGs: set of rules S -> NP VP This says that there are units called S, NP, and VP in this language That an S consists of an NP followed immediately by a VP Doesn’t say that’s the only kind of S Nor does it say that this is the only place that NPs and VPs occur LING 180 Autumn 2007 12

Generativity As with FSAs you can view these rules as either analysis or synthesis

Generativity As with FSAs you can view these rules as either analysis or synthesis machines Generate strings in the language Reject strings not in the language Impose structures (trees) on strings in the language How can we define grammatical vs. ungrammatical sentences? LING 180 Autumn 2007 13

Derivations A derivation is a sequence of rules applied to a string that accounts

Derivations A derivation is a sequence of rules applied to a string that accounts for that string Covers all the elements in the string Covers only the elements in the string LING 180 Autumn 2007 14

Derivations as Trees S NP VP NP Nom Pro Verb Det I prefer a

Derivations as Trees S NP VP NP Nom Pro Verb Det I prefer a Noun morning Noun flight LING 180 Autumn 2007 15

CFGs more formally A context-free grammar has 4 parameters (“is a 4 -tuple”) 1)

CFGs more formally A context-free grammar has 4 parameters (“is a 4 -tuple”) 1) A set of non-terminal symbols (“variables”) N 2) A set of terminal symbols (disjoint from N) 3) A set of productions P, each of the form – A -> – Where A is a non-terminal and is a string of symbols from the infinite set of strings ( N)* 4) A designated start symbol S LING 180 Autumn 2007 16

Defining a CF language via derivation A string A derives a string B if

Defining a CF language via derivation A string A derives a string B if A can be rewritten as B via some series of rule applications More formally: If A -> is a production of P and are any strings in the set ( N)* Then we say that – A directly derives or A Derivation is a generalization of direct derivation Let 1, 2, … m be strings in ( N)*, m>= 1, s. t. – – 1 2, 2 3… m-1 m We say that 1 derives m or 1* m We then formally define language LG generated by grammar G – – A set of strings composed of terminal symbols derived from S LG = {w | w is in * and S * w} LING 180 Autumn 2007 17

Parsing is the process of taking a string and a grammar and returning a

Parsing is the process of taking a string and a grammar and returning a (many? ) parse tree(s) for that string LING 180 Autumn 2007 18

Context? The notion of context in CFGs has nothing to do with the ordinary

Context? The notion of context in CFGs has nothing to do with the ordinary meaning of the word context in language All it really means is that the non-terminal on the left-hand side of a rule is out there all by itself (free of context) A -> B C Means that I can rewrite an A as a B followed by a C regardless of the context in which A is found LING 180 Autumn 2007 19

Key Constituents (English) Sentences Noun phrases Verb phrases Prepositional phrases LING 180 Autumn 2007

Key Constituents (English) Sentences Noun phrases Verb phrases Prepositional phrases LING 180 Autumn 2007 20

Sentence-Types Declaratives: A plane left S -> NP VP Imperatives: Leave! S -> VP

Sentence-Types Declaratives: A plane left S -> NP VP Imperatives: Leave! S -> VP Yes-No Questions: Did the plane leave? S -> Aux NP VP WH Questions: When did the plane leave? S -> WH Aux NP VP LING 180 Autumn 2007 21

NPs NP -> Pronoun I came, you saw it, they conquered NP -> Proper-Noun

NPs NP -> Pronoun I came, you saw it, they conquered NP -> Proper-Noun Los Angeles is west of Texas John Hennessy is the president of Stanford NP -> Det Noun The president NP -> Nominal -> Noun A morning flight to Denver LING 180 Autumn 2007 22

PPs PP -> Preposition NP From LA To the store On Tuesday morning With

PPs PP -> Preposition NP From LA To the store On Tuesday morning With lunch LING 180 Autumn 2007 23

Recursion We’ll have to deal with rules such as the following where the non-terminal

Recursion We’ll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly) NP -> NP PP VP -> VP PP [[The flight] [to Boston]] [[departed Miami] [at noon]] LING 180 Autumn 2007 24

Recursion Of course, this is what makes syntax interesting Flights from Flights from lunch

Recursion Of course, this is what makes syntax interesting Flights from Flights from lunch Denver Denver to to to Miami Miami in in February on a Friday under $300 with LING 180 Autumn 2007 25

Recursion [[Flights] [from Denver]] [to Miami]] [[[[Flights] [from Denver]] [to Miami]] [in February]] [on

Recursion [[Flights] [from Denver]] [to Miami]] [[[[Flights] [from Denver]] [to Miami]] [in February]] [on a Friday]] Etc. NP -> NP PP LING 180 Autumn 2007 26

Implications of recursion and context-freeness If you have a rule like VP -> V

Implications of recursion and context-freeness If you have a rule like VP -> V NP It only cares that the thing after the verb is an NP It doesn’t have to know about the internal affairs of that NP LING 180 Autumn 2007 27

The point VP -> V NP (I) hate flights from flights from lunch Denver

The point VP -> V NP (I) hate flights from flights from lunch Denver Denver to to to Miami Miami in in February on a Friday under $300 with LING 180 Autumn 2007 28

Bracketed Notation [S [NP [PRO I]] [VP [V prefer] [NP [Det a] [Nom [N

Bracketed Notation [S [NP [PRO I]] [VP [V prefer] [NP [Det a] [Nom [N morning] [N flight] ] ] S NP VP NP Nom Pro Verb Det I prefer a Noun morning Noun flight LING 180 Autumn 2007 29

Coordination Constructions S -> S and S John went to NY and Mary followed

Coordination Constructions S -> S and S John went to NY and Mary followed him NP -> NP and NP VP -> VP and VP … In fact the right rule for English is X -> X and X (Metarule) However we can say “He was longwinded and a bully. ” LING 180 Autumn 2007 30

Problems Agreement Subcategorization Movement (for want of a better term) LING 180 Autumn 2007

Problems Agreement Subcategorization Movement (for want of a better term) LING 180 Autumn 2007 31

Agreement This dog Those dogs *This dogs *Those dog This dog eats Those dogs

Agreement This dog Those dogs *This dogs *Those dog This dog eats Those dogs eat *This dog eat *Those dogs eats LING 180 Autumn 2007 32

Possible CFG Solution S -> NP VP NP -> Det Nominal VP -> V

Possible CFG Solution S -> NP VP NP -> Det Nominal VP -> V NP … Sg. S -> Sg. NP Sg. VP Pl. S -> Pl. Np Pl. VP Sg. NP -> Sg. Det Sg. Nom Pl. NP -> Pl. Det Pl. Nom Pl. VP -> Pl. V NP Sg. VP ->Sg. V Np … LING 180 Autumn 2007 33

CFG Solution for Agreement It works and stays within the power of CFGs But

CFG Solution for Agreement It works and stays within the power of CFGs But it’s ugly And it doesn’t scale all that well LING 180 Autumn 2007 34

Subcategorization Sneeze: John sneezed *John sneezed the book Say: You said [United has a

Subcategorization Sneeze: John sneezed *John sneezed the book Say: You said [United has a flight]S Prefer: I prefer [to leave earlier]TO-VP *I prefer United has a flight Give: Give [me]NP[a cheaper fare]NP Help: Can you help [me]NP[with a flight]PP *Give with a flight LING 180 Autumn 2007 35

Subcategorization Subcat expresses the constraints that a predicate (verb for now) places on the

Subcategorization Subcat expresses the constraints that a predicate (verb for now) places on the number and syntactic types of arguments it wants to take (occur with). LING 180 Autumn 2007 36

So? So the various rules for VPs overgenerate They permit the presence of strings

So? So the various rules for VPs overgenerate They permit the presence of strings containing verbs and arguments that don’t go together For example: VP -> V NP therefore Sneezed the book is a VP since “sneeze” is a verb and “the book” is a valid NP LING 180 Autumn 2007 37

Possible CFG Solution VP -> V NP PP … VP -> Intrans. V VP

Possible CFG Solution VP -> V NP PP … VP -> Intrans. V VP -> Trans. V NP VP -> Trans. Vw. PP NP PP … LING 180 Autumn 2007 38

Forward Pointer It turns out that verb subcategorization facts will provide a key element

Forward Pointer It turns out that verb subcategorization facts will provide a key element for semantic analysis (determining who did what to who in an event). LING 180 Autumn 2007 39

Movement Core example My travel agent booked the flight [[My travel agent]NP [booked [the

Movement Core example My travel agent booked the flight [[My travel agent]NP [booked [the flight]NP]VP]S i. e. “book” is a straightforward transitive verb. It expects a single NP arg within the VP as an argument, and a single NP arg as the subject. LING 180 Autumn 2007 40

Movement What about? Which flight do you want me to have the travel agent

Movement What about? Which flight do you want me to have the travel agent book? The direct object argument to “book” isn’t appearing in the right place. It is in fact a long way from where its supposed to appear. And note that it’s separated from its verb by 2 other verbs. LING 180 Autumn 2007 41

CFGs: a summary CFGs appear to be just about what we need to account

CFGs: a summary CFGs appear to be just about what we need to account for a lot of basic syntactic structure in English. But there are problems That can be dealt with adequately, although not elegantly, by staying within the CFG framework. There are simpler, more elegant, solutions that take us out of the CFG framework (beyond its formal power). Syntactic theories: HPSG, LFG, CCG, Minimalism, etc. LING 180 Autumn 2007 42

Other syntactic stuff Grammatical relations Subject – I booked a flight to New York

Other syntactic stuff Grammatical relations Subject – I booked a flight to New York – The flight was booked by my agent Object – I booked a flight to New York Complement – I said that I wanted to leave LING 180 Autumn 2007 43

Dependency parsing Word to word links instead of constituency Based on the European rather

Dependency parsing Word to word links instead of constituency Based on the European rather than American traditions But dates back to the Greeks The original notions of Subject, Object and the progenitor of subcategorization (called ‘valence’) came out of Dependency theory. Dependency parsing is quite popular as a computational model since relationships between words are quite useful LING 180 Autumn 2007 44

Dependency parsing S NP IN VBN NP NNS PP NP IN NN CC Parse

Dependency parsing S NP IN VBN NP NNS PP NP IN NN CC Parse tree: VP VBD PP NP NNS VP NNP Bills on ports and immigration were submitted by Senator Brownback Nesting of multi-word constituents submitted nsubjpass Bills prep_on ports auxpass were conj_and agent Brownback nn Senator Typed dep parse: Grammatical relations between individual words immigration LING 180 Autumn 2007 45

Why are dependency parses useful? Example: multi-document summarization Need to identify sentences from different

Why are dependency parses useful? Example: multi-document summarization Need to identify sentences from different documents that each say roughly the same thing: phrase structure trees of paraphrasing sentences which differ in word order can be significantly different but dependency representations will be very similar LING 180 Autumn 2007 46

Parsing: assigning correct trees to input strings Correct tree: a tree that covers all

Parsing: assigning correct trees to input strings Correct tree: a tree that covers all and only the elements of the input and has an S at the top For now: enumerate all possible trees A further task: disambiguation: means choosing the correct tree from among all the possible trees. LING 180 Autumn 2007 47

Treebanks Parsed corpora in the form of trees The Penn Treebank The Brown corpus

Treebanks Parsed corpora in the form of trees The Penn Treebank The Brown corpus The WSJ corpus Tgrep http: //www. ldc. upenn. edu/ldc/online/treebank/ Tregex http: //www-nlp. stanford. edu/nlp/javadoc/javanlp/ LING 180 Autumn 2007 48

Parsing involves search As with everything of interest, parsing involves a search which involves

Parsing involves search As with everything of interest, parsing involves a search which involves the making of choices We’ll start with some basic (meaning bad) methods before moving on to the one or two that you need to know LING 180 Autumn 2007 49

For Now Assume… You have all the words already in some buffer The input

For Now Assume… You have all the words already in some buffer The input isn’t pos tagged We won’t worry about morphological analysis All the words are known LING 180 Autumn 2007 50

Top-Down Parsing Since we’re trying to find trees rooted with an S (Sentences) start

Top-Down Parsing Since we’re trying to find trees rooted with an S (Sentences) start with the rules that give us an S. Then work your way down from there to the words. LING 180 Autumn 2007 51

Top Down Space S NP S VP S S NP Aux NP VP NP

Top Down Space S NP S VP S S NP Aux NP VP NP S VP VP S Aux NP VP S S VP VP NP LING 180 Autumn 2007 52

Bottom-Up Parsing Of course, we also want trees that cover the input words. So

Bottom-Up Parsing Of course, we also want trees that cover the input words. So start with trees that link up with the words in the right way. Then work your way up from there. LING 180 Autumn 2007 53

Bottom-Up Space Book that flight Book that flight LING 180 Autumn 2007 54

Bottom-Up Space Book that flight Book that flight LING 180 Autumn 2007 54

Control Of course, in both cases we left out how to keep track of

Control Of course, in both cases we left out how to keep track of the search space and how to make choices Which node to try to expand next Which grammar rule to use to expand a node LING 180 Autumn 2007 55

Top-Down, Depth-First, Left-to-Right Search LING 180 Autumn 2007 56

Top-Down, Depth-First, Left-to-Right Search LING 180 Autumn 2007 56

Example LING 180 Autumn 2007 57

Example LING 180 Autumn 2007 57

Example [flight] LING 180 Autumn 2007 58

Example [flight] LING 180 Autumn 2007 58

Example flight LING 180 Autumn 2007 59

Example flight LING 180 Autumn 2007 59

Top-Down and Bottom-Up Top-down Only searches for trees that can be answers (i. e.

Top-Down and Bottom-Up Top-down Only searches for trees that can be answers (i. e. S’s) But also suggests trees that are not consistent with the words Bottom-up Only forms trees consistent with the words Suggest trees that make no sense globally LING 180 Autumn 2007 60

So Combine Them There a million ways to combine top-down expectations with bottom-up data

So Combine Them There a million ways to combine top-down expectations with bottom-up data to get more efficient searches Most use one kind as the control and the other as a filter As in top-down parsing with bottom-up filtering LING 180 Autumn 2007 61

Adding Bottom-Up Filtering LING 180 Autumn 2007 62

Adding Bottom-Up Filtering LING 180 Autumn 2007 62

3 problems with TDDFLt. R Parser Left-Recursion Ambiguity Inefficient reparsing of subtrees LING 180

3 problems with TDDFLt. R Parser Left-Recursion Ambiguity Inefficient reparsing of subtrees LING 180 Autumn 2007 63

Left-Recursion What happens in the following situation S -> NP VP S -> Aux

Left-Recursion What happens in the following situation S -> NP VP S -> Aux NP VP NP -> NP PP NP -> Det Nominal … With the sentence starting with – Did the flight… LING 180 Autumn 2007 64

Ambiguity “One morning I shot an elephant in my pajamas. How he got into

Ambiguity “One morning I shot an elephant in my pajamas. How he got into my pajamas I don’t know. ” Groucho Marx LING 180 Autumn 2007 65

Lots of ambiguity VP -> VP PP NP -> NP PP Show me the

Lots of ambiguity VP -> VP PP NP -> NP PP Show me the meal on flight 286 from SF to Denver 14 parses! LING 180 Autumn 2007 66

Lots of ambiguity § Church and Patil (1982) Number of parses for such sentences

Lots of ambiguity § Church and Patil (1982) Number of parses for such sentences grows at rate of number of parenthesizations of arithmetic expressions Which grow with Catalan numbers PPs 1 2 3 4 5 6 Parses 2 5 14 132 469 1430 LING 180 Autumn 2007 67

Avoiding Repeated Work Parsing is hard, and slow. It’s wasteful to redo stuff over

Avoiding Repeated Work Parsing is hard, and slow. It’s wasteful to redo stuff over and over. Consider an attempt to top-down parse the following as an NP: A flight from Indianapolis to Houston on TWA LING 180 Autumn 2007 68

flight LING 180 Autumn 2007 69

flight LING 180 Autumn 2007 69

flight LING 180 Autumn 2007 70

flight LING 180 Autumn 2007 70

flight LING 180 Autumn 2007 71

flight LING 180 Autumn 2007 71

flight LING 180 Autumn 2007 72

flight LING 180 Autumn 2007 72

Grammars and Parsing Context-Free Grammars and Constituency Some common CFG phenomena for English Baby

Grammars and Parsing Context-Free Grammars and Constituency Some common CFG phenomena for English Baby parsers: Top-down and Bottom-up Parsing Thursday: Real parsers: : Dynamic Programming parsing CKY Probabilistic parsing Optional section: the Earley algorithm LING 180 Autumn 2007 73