Natural Language Processing Language is communication Christopher Manning

Natural Language Processing “Language is communication” Christopher Manning ❀ @chrmanning Stanford University ❀ @stanfordnlp

Can my computer set an alarm for me?

Can my computer set an alarm for me? Can it tell whether a restaurant has a vegetarian menu?

Can my computer set an alarm for me? Can it tell whether a restaurant has a vegetarian menu? Can it book me a vacation in Fiji?

Boy paralyzed after tumor fights back to gain black belt

Boy paralyzed after [tumor fights back to gain black belt]

Boy [paralyzed after tumor] fights back to gain black belt

Machine-learning language

Stanford Core. NLP is one of the most used NLP tools in the world Academics, companies, governments, and non-profits Tens of thousands of users of open source version Dozens of licensed commercial users

From research to industrial-strength tools for language analysis: Stanford Core. NLP

From research to industrial-strength tools for language analysis: Stanford Core. NLP

Co. NLL 2017 Shared Task: Universal Dependencies Parsing 1. Stanford (Stanford) 76. 30 ± 0. 12 2. C 2 L 2 (Ithaca) 75. 00 ± 0. 12 3. IMS (Stuttgart) 74. 42 ± 0. 13 4. HIT-SCIR (Harbin) 72. 11 ± 0. 14 5. LATTICE (Paris) 70. 93 ± 0. 13 6. NAIST SATO (Nara) 70. 14 ± 0. 13 . . . http: //universaldependencies. org/conll 17/results. html

What will drive NLP breakthroughs during the next decade? Towards precise Natural Language Understanding

Word distributions distributed word representations We expect 1997 sales growth for the Many traders don’t expect the FOMC to You might expect it to resemble a UFO Tourists expect faster clearance processes Fire officials don’t expect to douse the flames expect = 0. 286 0. 792 − 0. 177 − 0. 107 0. 109 − 0. 542 0. 349 0. 271 0. 487 [Bengio et al. 2003, Mnih & Hinton 2008, Collobert & Weston 2008, Turian 2010, Mikolov 2013, Pennington 2014, etc. ]

Glo. Ve: A new model for learning word representations [Pennington et al. , EMNLP 2014] Crucial insight: Ratios of co-occurrence probabilities can encode meaning components! x = solid x = gas x = water x = random large small large small ~1 ~1

Word similarities http: //nlp. stanford. edu/projects/glove/ Nearest words to frog: 1. frogs 2. toad 3. litoria 4. leptodactylidae 5. rana 6. lizard 7. eleutherodactylus litoria leptodactylidae rana eleutherodactylus

Named Entity Recognition Performance Model on Co. NLL Categorical CRF SVD (log tf) Collobert&W Glo. Ve (us!) Co. NLL 2003 dev Co. NLL 2003 test 91. 0 90. 5 92. 2 93. 2 85. 4 84. 8 87. 4 88. 3 F 1 score of CRF trained on Co. NLL 2003 English with 50 dim word vectors.

Named Entity Recognition Performance Model on Co. NLL Categorical CRF SVD (log tf) Collobert&W Glo. Ve (us!) Co. NLL 2003 dev Co. NLL 2003 test 91. 0 90. 5 92. 2 93. 2 85. 4 84. 8 87. 4 88. 3 F 1 score of CRF trained on Co. NLL 2003 English with 50 dim word vectors. ACE 2 MUC 7 77. 4 73. 6 81. 7 82. 9 73. 4 71. 5 80. 2 82. 2

Goals for Natural Language Understanding Supporting human-human communication: Can we solve the language translation barrier?

Neural Machine Translation [Sutskever et al. 2014, Bahdanau et al. 2014, Luong and Manning 2015] The Sentence meaning is built up Source sentence protests escalated over the weekend <EOS> 0. 1 0. 3 0. 1 -0. 4 0. 2 0. 6 -0. 1 -0. 7 0. 1 0. 4 0. 3 -0. 2 -0. 3 0. 5 0. 9 -0. 3 -0. 2 0. 6 -0. 1 -0. 5 0. 1 -0. 1 0. 6 -0. 1 -0. 7 0. 1 0. 2 0. 6 -0. 1 -0. 7 0. 1 0. 3 0. 6 -0. 1 -0. 7 0. 1 0. 4 -0. 1 -0. 7 0. 1 -0. 2 0. 6 -0. 1 -0. 7 0. 1 -0. 4 0. 6 -0. 1 -0. 7 0. 1 -0. 3 0. 5 -0. 1 -0. 7 0. 1 0. 2 -0. 1 0. 2 0. 6 -0. 1 -0. 7 0. 1 0. 3 -0. 1 -0. 7 0. 1 0. 2 0. 6 -0. 1 -0. 4 0. 1 0. 2 -0. 8 -0. 1 -0. 5 0. 1 0. 2 0. 6 -0. 1 -0. 7 0. 1 0. 2 -0. 1 -0. 7 0. 1 -0. 4 0. 6 -0. 1 -0. 7 0. 1 0. 2 0. 6 -0. 1 0. 3 0. 1 -0. 1 0. 6 -0. 1 0. 3 0. 1 0. 2 0. 4 -0. 1 0. 2 0. 1 0. 3 0. 6 -0. 1 -0. 5 0. 1 0. 2 0. 6 -0. 1 -0. 7 0. 1 0. 4 -0. 6 0. 2 -0. 3 0. 4 0. 2 -0. 3 -0. 1 -0. 4 0. 2 0. 4 0. 1 -0. 5 -0. 2 0. 4 -0. 2 -0. 3 -0. 4 -0. 2 0. 6 -0. 1 -0. 7 0. 1 -0. 1 0. 3 -0. 1 -0. 7 0. 1 -0. 2 0. 6 0. 1 0. 3 0. 1 -0. 4 0. 5 -0. 5 0. 4 0. 1 0. 2 0. 6 -0. 1 -0. 7 0. 1 Die Proteste waren am Wochenende eskaliert <EOS> The protests escalated over the weekend Translation generated

Stanford NMT 2015: Attention (Luong et al. EMNLP 2015; Luong & Manning IWSLT 2015)

IWSLT 2015, TED talk MT, English-German BLEU (cased) HTER (HE set) 35 30 28. 18 30. 85 30 26. 18 25 26. 02 25 24. 96 22. 51 20. 08 20 21. 84 22. 67 23. 42 20 16. 16 15 15 10 10 24 26 % 5 5 0 0 Stanford Karlsruhe Edinburgh Heidelberg PJAIT Baseline Stanford Edinburgh Karlsruhe Heidelberg PJAIT

Goals for Natural Language Understanding Getting what humans mean: Interpreting tone and nuance

Sentiment analysis Is someone expressing positive, negative, or neutral views? Sometimes easy looking at a “bag of words” … … loved … … … great … … … impressed … … entertaining … But often it’s more subtle

Tree-structured Long Short. Term Memory (LSTM) Generalizes sequential LSTM to trees with any branching factor

Tree-structured Long Short-Term Memory (LSTM) Sentiment Analysis [Tai, Socher & Manning 2015] A tree-structured network can capture contrastive sentences like X but Y

Goals for Natural Language Understanding Knowledge-aware, flexible, robust, and effective dialogue agents Not just a chatbot, not an inflexible dialogue tree

Effective and flexible understanding using knowledge (Eric et al. 2017) Hi, what can I help you with? I feel like Italian There are 3 italian restaurants nearby Hmm … actually any froyo around? No froyo, sadly, but there is a gelato cafe Cool! Where is it?

A deep learning model combining knowledge, short term memory, and language generation

Goals for Natural Language Understanding Learning on the job to work with human beings

Interactive language learning Voice interfaces are starting to appear everywhere Should thousands of engineers be hand-writing “skills” for each? How can we get them to understand us the way we talk?

Interactive Language Learning Can a computer and person learn to collaborate?

Goals • Interactive learning • receive feedback from users, improve through use • Adapt to users • regular weekday alarm, call my children • Handle special domains and low resource languages • familiar words take on new meanings • revert to commit 25 ad 3 • push to torch master

Experiments: Examples of exchanges (good)

Experiments: Examples of exchanges (interesting)

Calendar scheduling and updating Now applying to more realistic and useful tasks. Dialogue agents for: • calendaring • graphs/charts

Goals for Natural Language Understanding text to be able to give information

Open-domain Question Answering (Chen, et al. ACL 2017) https: //arxiv. org/abs/1704. 00051 Q: How many of Warsaw's inhabitants spoke Polish in 1933? Document Retriever Document Reader 833, 500

Document Reader: Stanford Attentive Reader Q (Chen et al, 2016) characters in " @placeholder " movies have gradually become more diverse Bidirectional LSTMs … … … characters in “ @placeholder more diverse

Stanford Attentive Reader++ Q Who did Genghis Khan unite before he began conquering the rest of Eurasia? Bidirectional RNNs P … Attention 43 predict start token … … Attention predict end token

(Chen et al, 2017) SQUAD Dataset Results (single model) F 1 Logistic regression 51. 0 Fine-Grained Gating (Carnegie Mellon U) 73. 3 Match-LSTM (Singapore Management U) 73. 7 DCN (Salesforce) 75. 9 Bi. DAF (UW & Allen Institute) 77. 3 Multi-Perspective Matching (IBM) 78. 7 Reaso. Net (MSR Redmond) 79. 4 Dr. QA (Chen et al. 2017) 79. 4 r-net (MSR Asia) 80. 8 Human performance 91. 2

The Stanford NLP Group Dan Jurafsky, Percy Liang, Christopher Manning & about 40 students and postdocs