A Machine Learning Approach to Coreference Resolution of

A Machine Learning Approach to Coreference Resolution of Noun Phrases 12/22/2021 1

Outline The notion of Coreference l A Machine learning approach l Extraction of Markables ¡ Extracted Features ¡ Training Data ¡ Classifier Construction ¡ Testing ¡ l Result analysis 12/22/2021 2 2

The notion of Coreference Definition The grammatical relation between two words that have a common referent (Word. Net) l In linguistics, Coreference is the phenomenon where two expressions in an utterance both refer to the same thing (Wikipedia) l A Coreference resolution process output pairs of noun phrases (coreferences) l 12/22/2021 3 5

A typical example of anaphoric expression are pronouns such as he in the text l John arrived. He looked tired. l 12/22/2021 4

The notion of Coreference Usage Information Retrieval l Question answering l Shallow parsing l And more… l 12/22/2021 5 6

The notion of Coreference Example (Eastern Air)a 1 Proposes (Date For Talks on ((Pay)c 1 -Cut)d 1 Plan)b 1. (Eastern Airlines)a 2 executives noticed (union)e 1 leaders that the carrier wishes to discuss selective ((wage)c 2 reductions)d 2 on (Feb. 3)b 2. ((Union)e 2 representatives who could be reached)f 1 said (they)f 2 hadn’t decided whether (they)f 3 would respond. By proposing (a meeting date)b 3, (Eastern)a 3 moved one step closer toward reopening current high-cost contract agreements with ((its)a 4 unions)e 3. 12/22/2021 6 10

Outline The notion of Coreference l A Machine learning approach l Extraction of Markables ¡ Extracted Features ¡ Training Data ¡ Classifier Construction ¡ Testing ¡ l Result analysis 12/22/2021 7 11

Extraction of Markables Preprocessing 12/22/2021 8 14

Outline The notion of Coreference l A Machine learning approach l Extraction of Markables ¡ Extracted Features ¡ Training Data ¡ Classifier Construction ¡ Testing ¡ l Result analysis 12/22/2021 9 15

Extracted Features l 12 suggested features for markables pairs Distance (How far the two markables are) ¡ i/j is a Pronoun (he, himself, his…) ¡ String match feature (base strings match) ¡ j is a Definite noun phrase (the) ¡ j is a Demonstrative noun phrase (this, that, these, those) ¡ Number agreement (i and j are both plural/singular) ¡ 12/22/2021 10 19

Extracted Features cont. l 12 suggested features for markables pairs Semantic class agreement (i and j are of the same Word. Net class) ¡ Gender agreement (i and j are of the same gender) ¡ Both proper name (i and j are proper names) ¡ Alias (i and j match. e. g. 1 st jan and 01. 01 for dates) ¡ Apposition (j is an apposition of i. e. g. Mubarak, Egypt's president) ¡ 12/22/2021 11 22

Extracted Features Example 12/22/2021 12 25

Outline The notion of Coreference l A Machine learning approach l Extraction of Markables ¡ Extracted Features ¡ Training Data ¡ Classifier Construction ¡ Testing ¡ l Result analysis 12/22/2021 13 26

Training Data MUC-6/7 conference corpora l Creating positive examples l Creating negative examples l 12/22/2021 14 27

Outline The notion of Coreference l A Machine learning approach l Extraction of Markables ¡ Extracted Features ¡ Training Data ¡ Classifier Construction ¡ Testing ¡ l Result analysis 12/22/2021 15 28

Classifier Construction l Classifier types: neural network, SVM, KNN, Decision tree (selected) l Decision tree structure: Each node of the tree is a question about one of the features. ¡ According to the answer, the path is chosen. ¡ When a leaf is reached, its label is returned. ¡ 12/22/2021 16 31

Outline The notion of Coreference l A Machine learning approach l Extraction of Markables ¡ Extracted Features ¡ Training Data ¡ Classifier Construction ¡ Testing ¡ l Result analysis 12/22/2021 17 32

Testing After a classifier is built, it is tested against a preannotated example. l Then, the results are compared with the “true” anotation. l The measures are Recall (how many of the real coreferences were returned) and Precision (how many of the coreferences returned, are true ones). l 12/22/2021 18 34

Testing Example (Ms. Washington)73's candidacy is being championed by (several powerful lawmakers)74 including ((her)76 boss)75, Chairman John Dingell)77 (D. , (Mich. )78) of (the House Energy and Commerce Committee)79. (She)80 currently is (a counsel)81 to (the committee)82. (Ms. Washington)83 and (Mr. Dinge. U)84 have been considered (allies)85 of (the (securities)87 exchanges)86, while (banks)88 and ((futures)90 exchanges)89 have often fought with (them)91. 12/22/2021 19 37

Testing Example Classification 12/22/2021 20 40

Outline The notion of Coreference l A Machine learning approach l Extraction of Markables ¡ Extracted Features ¡ Training Data ¡ Classifier Construction ¡ Testing ¡ l Result analysis 12/22/2021 21 41

Result analysis Decision Tree 12/22/2021 22 44

Result analysis Recall & Precision 12/22/2021 23 45

Result analysis misconceptions The Decision tree shows that only 8 features are being used. l When used with 3 features (alias, apposition, string match) the scores (f-measure) were only 1 -2. 3% worse then when used with all of them only 3 features really contribute. l 12/22/2021 24 47

Result analysis misconceptions – cont. 66. 3% of the positive results followed the path of the first tree node – string matching. l 70% of the total precision problems are caused by string matching: l ¡ 12/22/2021 Directors also approved the election of Allan Laufgraben, 54 years old, as president and (chief executive officer)1 and Peter A. Left, 43, as chief operating officer. Milton Petrie, 90 -year-old chairman, president and (chief executive officer)2 since the company was founded in 1932, will continue as chairman. 25 49

Result analysis conclusions The great achievement according to the authors – the fact that a learning method, over “shallow features” achieves the same performance as top-ofthe-art systems. l A HUGE majority of the results (and errors) is determined by 1 -3 features. l Learning over such a small amount of features isn’t really learning. So the achievement does not look like one. Not to me, though. l 12/22/2021 26 52