Parsing German with Latent Variable Grammars Slav Petrov

Parsing German with Latent Variable Grammars Slav Petrov and Dan Klein UC Berkeley

The Game of Designing a Grammar § Annotation refines base treebank symbols to improve statistical fit of the grammar § Parent annotation [Johnson ’ 98] § Head lexicalization [Collins ’ 99, Charniak ’ 00] § Automatic clustering?

Previous Work: Manual Annotation [Klein & Manning ’ 03] § Manually split categories § NP: subject vs object § DT: determiners vs demonstratives § IN: sentential vs prepositional § Advantages: § Fairly compact grammar § Linguistic motivations § Disadvantages: § Performance leveled out § Manually annotated Model F 1 Naïve Treebank Grammar 72. 6 Klein & Manning ’ 03 86. 3

[Matsuzaki et. al ’ 05, Prescher ’ 05] Previous Work: Automatic Annotation Induction § Advantages: § Automatically learned: Label all nodes with latent variables. Same number k of subcategories for all categories. § Disadvantages: § Grammar gets too large § Most categories are oversplit while others are undersplit. Model Klein & Manning ’ 03 Matsuzaki et al. ’ 05 F 1 86. 3 86. 7

Overview § Learning: § Hierarchical Training § Adaptive Splitting § Parameter Smoothing § Inference: § Coarse-To-Fine Decoding § Variational Approximation § German Analysis [Petrov, Barrett, Thibaux & Klein in ACL’ 06] [Petrov & Klein in NAACL’ 07]

Learning Latent Annotations Forward EM algorithm: § Brackets are known § Base categories are known § Only induce subcategories X 1 X 2 X 3 X 7 X 4 X 5 X 6. Just like Forward-Backward for HMMs. He was right Backward

Starting Point Limit of computatio nal resources

Refinement of the DT tag DT DT-1 DT-2 DT-3 DT-4

Refinement of the DT tag DT

Hierarchical Refinement of the DT tag DT

Hierarchical Estimation Results Model F 1 Baseline 87. 3 Hierarchical Training 88. 4

Refinement of the , tag § Splitting all categories the same amount is wasteful:

The DT tag revisited Oversplit?

Adaptive Splitting § Want to split complex categories more § Idea: split everything, roll back splits which were least useful

Adaptive Splitting § Want to split complex categories more § Idea: split everything, roll back splits which were least useful

Adaptive Splitting § Evaluate loss in likelihood from removing each split = Data likelihood with split reversed Data likelihood with split § No loss in accuracy when 50% of the splits are reversed.

Adaptive Splitting Results Model F 1 Previous 88. 4 With 50% Merging 89. 5

Number of Phrasal Subcategories

Number of Lexical Subcategories

Smoothing § Heavy splitting can lead to overfitting § Idea: Smoothing allows us to pool statistics

Linear Smoothing

Result Overview Model F 1 Previous 89. 5 With Smoothing 90. 7

[Goodman ‘ 97, Charniak&Johnson ‘ 05] Coarse-to-Fine Parsing Treebank Parse Coarse grammar Prune NP … VP Parse NP-apple NP-1 VP-run VP-6 NP-17 NP-12 … VP-31 NP-dog NP-cat … NP-eat … … grammar Refined grammar

Hierarchical Pruning Consider the span 5 to 12: coarse: split in two: split in four: split in eight: … … … QP NP VP … QP 1 QP 2 NP 1 NP 2 VP 1 VP 2 … … QP 1 QP 3 QP 4 NP 1 NP 2 NP 3 NP 4 VP 1 VP 2 VP 3 VP 4 … … … … …

Intermediate Grammars X-Bar=G 0 DT G 1 Learning G 2 G 3 G 4 G 5 G= G 6 DT 1 DT 2 DT 3 DT 4 DT 1 DT 2 DT 3 DT 4 DT 5 DT 6 DT 7 DT 8

State Drift (DT tag) the that this That This …… That …… some this these some this …… these that …… some …… EM

Projected Grammars G 1 1(G) G 2 2(G) G 3 3(G) G 4 4(G) G 5 5(G) G= G 6 G Projection i 0(G) Learning X-Bar=G 0

Bracket Posteriors (after G 0)

Bracket Posteriors (after G 1)

(Movie) Chart) Bracket Posteriors (Final

Bracket Posteriors (Best Tree)

Parse Selection Parses: Derivations: -1 -1 -1 -2 -2 -1 -1 -2 Computing most likely unsplit tree is NP-hard: § Settle for best derivation. § Rerank n-best list. § Use alternative objective function / Variational Approximation.

Efficiency Results § Berkeley Parser: § 15 min § Implemented in Java § Charniak & Johnson ‘ 05 Parser § 19 min § Implemented in C

Accuracy Results all F 1 ENG Charniak&Johnson ‘ 05 (generative) 90. 1 89. 6 This Work 90. 6 90. 1 GER Dubey ‘ 05 76. 3 - This Work 80. 8 80. 1 Chiang et al. ‘ 02 80. 0 76. 6 This Work 86. 3 83. 4 CHN ≤ 40 words F 1

Parsing German Shared Task § Two Pass Parsing § Determine constituency structure (F 1: 85/94) § Assign grammatical functions § One Pass Approach § Treat categories+grammatical functions as labels

Parsing German Shared Task § Two Pass Parsing § Determine constituency structure § Assign grammatical functions § One Pass Approach § Treat categories+grammatical functions as labels

Development Set Results

Shared Task Results

Part-of-speech splits

Linguistic Candy

Conclusions § Split & Merge Learning § Hierarchical Training § Adaptive Splitting § Parameter Smoothing § Hierarchical Coarse-to-Fine Inference § Projections § Marginalization § Multi-lingual Unlexicalized Parsing

Thank You! Parser is avaliable at http: //nlp. cs. berkeley. edu