A Generative Model for Punctuation in Dependency Parsing

A Generative Model for Punctuation in Dependency Parsing Xiang Lisa Li*, Dingquan Wang*, and Jason Eisner * Equal Contribution 1

NLP has neglected punctuation Treebanks treat punctuation marks as ordinary tokens, but ARE THEY? The Linguistics of Punctuation, Nunberg (1990) 2

Punctuation is useful Punctuation marks are correlated with prosody and the syntactic tree structure. Tree parse generation Punctuation 3

. . . root Hail dobj king appos the wields rel-clause Pendragon dobj who Arthur , , 4 Excalibur . . . ,

Point Absorption root Quote Transposition dobj Geoffrey Nunberg appos rel-clause dobj Hail the king , Arthur Pendragon , , 5 , , who wields `` Excalibur`` , ? . . ? , , . ````. . ? ,

Summary… • Punctuation marks are not words. • • They do not belong in the tree. Only underlying punctuation marks are in the tree, where they surround certain phrases. • Just as prosody is not words. 7

Tree parse generation underlying punct Punctuation surface punct The surface punctuation has a non-obvious correlation with the tree. The underlying punctuation has a more direct correlation with the tree. 9

Tree Attach parse underlying punct generation Rewriting surface punct Let’s exploit the underlying punctuation under a generative model ! 10

P(punctuated tree | unpunctuated tree) = P( |root, unpunctuated tree) . Attach . . root dobj appos rel-clause dobj Hail the king Arthur Pendragon 11 who wields Excalibur

P(punctuated tree | unpunctuated tree) = P( |root, unpunctuated tree) * P( |appos, unpunctuated tree) . root . , , Attach dobj , , , appos , rel-clause dobj Hail the king Arthur Pendragon 12 who wields Excalibur .

P(punctuated tree | unpunctuated tree) = P( |root, unpunctuated tree) * P( |appos, unpunctuated tree) * P( |rel-clause, unpunctuated tree) . root . Attach , , dobj , appos , , , rel-clause dobj Hail the king , Arthur Pendragon 13 , who wields Excalibur .

P(punctuated tree | unpunctuated tree) = P( |root, unpunctuated tree) * P( |appos, unpunctuated tree) * P( |rel-clause, unpunctuated tree) * P( |dobj, unpunctuated tree) . root . dobj Attach , , ```` , appos , , , rel-clause `` ` `` dobj Hail the king , Arthur Pendragon 14 , , who wields Excalibur , .

P(punctuated tree | unpunctuated tree) = P( |root, unpunctuated tree) * P( |appos, unpunctuated tree) * P( |rel-clause, unpunctuated tree) * P( |dobj, unpunctuated tree) . root . dobj , , ```` , appos , , , rel-clause Hail the king Attach , Arthur Pendragon 15 `` dobj `` , , who wields `` Excalibur`` , .

appos left right P( , , | Prob punct , , 0. 4 `` `` 0. 3 , 0. 05 0. 2 • • appos, unpunctuated tree) = 0. 4 f 1 f 2 f 3 f 4 0 0 1 1 1 0 0 0 2 0 0 1 0 1 1 0 f 1: indicator feature for no punctuation ( ). f 2: indicator feature that checks for symmetry. f 3: span length feature for attaching commas. f 4: indicator feature that checks for internal comma. 16 . . .

P(surface punct | punctuated tree) Rewriting root . dobj , appos , , , rel-clause Hail the king , , `` dobj `` , , who wields `` Excalibur`` , . Arthur Pendragon , who wields `` Excalibur. `` Arthur Pendragon 17

Rewriting — A Sliding Window Model Left-absorb , , , ) ``. Left-absorb identity , , , P( , ) ) ) P( ) ) ``. P( `` `` P( swap 20 ) ) Geoffrey Nunberg ) )

Summary generative story • Underlying punctuation is generated at each tree node (not quite independently). • Total underlying punctuation at each slot between words is rewritten into surface punctuation (independently). 22

Tree infer parse underlying punct generation surface punct In training, we observe the tree and surface punct. Want to recover the underlying punctuation. P(underlying punct | tree, surface punct) 23

. . . Method 1. A Sampling Approach root . `` P(underlying punct| tree, surface punct) P(underlying | tree) * P(surface | underlying) dobj `` , appos , , , rel-clause Hail the king , , `` dobj `` , , who wields `` Excalibur`` , . Arthur Pendragon , who wields `` Excalibur. `` Arthur Pendragon 24

. . . Method 1. A Sampling Approach root . `` P(underlying punct| tree, surface punct) P(underlying | tree) * P(surface | underlying) dobj `` , appos , , , rel-clause Hail the king , , `` dobj `` , , who wields `` Excalibur`` , . Arthur Pendragon , who wields `` Excalibur. `` Arthur Pendragon 25

. . . Method 1. A Sampling Approach root . P(underlying punct| tree, surface punct) P(underlying | tree) * P(surface | underlying) dobj , appos , , , rel-clause Hail the king , , `` dobj `` , , who wields `` Excalibur`` , . Arthur Pendragon , who wields `` Excalibur. `` Arthur Pendragon 26

Method 2. O(n) Dynamic Programming Idea : Make a (weighted) CFG that generates just the possible un Soft constraint 1 : Underlying punctuation fits the tree structure Soft constraint punctuation Intersect 2 1 : Underlying punctuation fits surface 2 our CFG 28

Method 2. O(n) Dynamic Programming Idea : Make a (weighted) CFG that generates just the possible un Soft constraint 1 : Underlying punctuation fits the tree structure PCFG Soft constraint punctuation Intersect 2 1 : Underlying punctuation fits surface 2 our CFG 29

Method 2. O(n) Dynamic Programming Idea : Make a (weighted) CFG that generates just the possible un Soft constraint 1 : Underlying punctuation fits the tree structure PCFG Soft constraint punctuation 2 : Underlying punctuation fits surface weighted FSA ？ Hail the king Intersect , ？ Arthur Pendragon 1 2 , ？ who wields our CFG 30 `` ？ Excalibur . ``

Method 2. O(n) Dynamic Programming Idea : Make a (weighted) CFG that generates just the possible un Soft constraint 1 : Underlying punctuation fits the tree structure PCFG Soft constraint punctuation Intersect PCFG 2 1 : Underlying punctuation fits surface 2 weighted FSA our CFG weighted FSA 31 = weighted CFG

Method 2. O(n) Dynamic Programming Idea : Make a (weighted) CFG that generates just the possible un Soft constraint 1 : Underlying punctuation fits the tree structure Soft constraint punctuation Intersect 2 1 : Underlying punctuation fits surface 2 our CFG Now we can find the best underlyingly punctuated tree • or sum over all of them for the model likelihood in training For our CFG, this can be done in O(n) time 32

Results Rules Learned from the Noisy Channel Analysis of Attachment model Punctuation Restoration Syntactic Rephrasing 35

We learn high probability for Nunberg’s English Rules American English British English , , underlying surface , , , . , Distribution over rewrite rules 36 . `` `` . . ``

Viterbi recovers good underlying punctuation root , . Real Examples (simplified for slide) , advmod , Earlier , Kerry , , ``, in said , `` in said nmod , , fact , fact 38 , `` ccomp `` dobj ``. question. `` answered the question answered the

Viterbi recovers good underlying punctuation root Real Examples (simplified for slide) . Oxford Comma dobj , Section 1 , , conj 2 2 , , , 39 conj 7 7 , conj , , and 8 . .

English Spanish Arabic Chinese Hindi What if we don’t have gold parses? Punctuation Restoration average edit distance [the 40 lower the better] Bi. LSTM-CRF tagger Always Period tagger Ours

root conj punct but , advcl if true punct , punct dobj the caper failed Syntactically Transform Sentence from Treebank 43 .

root dobj conj but the caper failed punct . , advcl punct if true obviously ugly, and high perplexity under an LM Syntactically Transform Sentence from Treebank 44 ,

root conj punct but , , advcl if true , punct dobj the caper . failed . Syntactically Transform Sentence with our Annotation 45

root conj , but if true advcl , . dobj the caper failed Syntactically Transform Sentence with our Annotation 46

root . , dobj conj but the caper failed , if but the caper failed , advcl true if true , , . . Syntactically Transform Sentence with our Annotation 47

48

Implemented by Finite State Transducer 49

Implemented by Finite State Transducer 50

51

Implemented by Finite State Transducer 52