Model of Prefrontal Cortex for Language Processing and

Model of Prefrontal Cortex for Language Processing and Human-Robot Interaction Xavier HINAUT INRIA La. BRI Institut des Maladies Neurodégénératives Bordeaux, France xavier. hinaut@inria. fr www. xavierhinaut. com github. com/neuronal. X

Model the brain as a synergy of memories

Abstract sequence processing in brains, machines & robots Enaction Neuroscience Modelling Robotics Developmental Learning Machine Learning Xavier HINAUT - Mnémosyne 4

How the brain deals with (complex) sequences? • Is there a generic architecture for – Representing the order of elements in a sequence? – Memorizing some elements for a long period of time • gating mechanism / working memory – Learning to recognize/generate sensori-motor sequences: • Birdsong • Human language • Abstract sequence of movements • How to give more “human-like” language processing to robots? Xavier HINAUT - Mnémosyne 5

More “human-like” language processing for robots • Parse complex sentences (not stereotypical) – E. g. “Grasp the red box beside the green cup” • Understand a sentence during the sentence – Anytime algorithm – Quicker interaction & turn-taking • Could enable to prepare next action • Deal with noisy speech recognition Xavier HINAUT - Mnémosyne 6

Don’t unfold time, please! Processing Time Recurrent Neural Net. Processing Time Feed Forward Neural Net. Sliding window Xavier HINAUT - Mnémosyne 7

Don’t unfold time, please! Still, back-propagation can be used with RNNs. . . Unfold time Duplicate the network for each time step. Xavier HINAUT - Mnémosyne 8

Generic Neural Circuit for Abstract Sequence Processing? Striatum PFC Not unfolding time like other RNN algorithms Xavier HINAUT - Mnémosyne 9

Reservoir Computing Striatum PFC Prefrontal Cortex (PFC) Striatum Xavier HINAUT - Mnémosyne Goldman-Rakic, 1987. Dominey 1995. Hinaut et al. 2011, 2013 10

Reservoir Computing Striatum PFC Prefrontal Cortex (PFC) Striatum Not unfolding time like other RNN algorithms Xavier HINAUT - Mnémosyne 11

Reservoir Computing for Cortico-basal modelling Dominey 2009 Dominey 1995 Mannella & Baldassarre 2015 Xavier HINAUT - Mnémosyne 12

Reservoir Computing Win W Wout • Win, W: – randomly generated • Wout : – learned Jaeger 2001, Jaeger et al. 2007, Lukosevicius 2012 Xavier HINAUT - Inria, Bordeaux, France 13

Abstract sequence processing PLo. S ONE 2013 & ICANN 2012 + 2014 J. of Physiology-Paris 2011 Front. In Neuro. Robotics 2014 & Brain & Language (in revision) Xavier HINAUT - Mnémosyne (in preparation) 14

Abstract sequence processing A, B = Turn, Push, Pull X, Y, Z = girl, hit, boy AABB The X Y the Z ABAB The Y was X by the Z Hinaut et al. 2013, 2014, 2015 Hinaut & Dominey 2011 On the Z X the Y X (Y, Z) X, Y, Z = put, toy, left Hinaut et al. 2014, 2015 Xavier HINAUT - Mnémosyne (in preparation) 15

[…] He took his vorpal sword in hand: Long time the manxome foe he sought— So rested he by the Tumtum tree, And stood awhile in thought. […] Xavier HINAUT - Inria, Bordeaux, France 16

[…] He took his vorpal sword in hand: Long time the manxome foe he sought— So rested he by the Tumtum tree, And stood awhile in thought. […] Jabberwocky, Lewis Carroll Through the Looking-Glass, and What Alice Found There (1871) Xavier HINAUT - Inria, Bordeaux, France 17

Task: Thematic Role Assignment – Representation of meaning (predicate form) : Predicate ( Agent , Object, Recipient ) « The boy gives the ball to the dog » Give ( Boy , Ball , Dog ) « The ball is given by the boy to the dog » Mapping between semantic words and their roles ( P ( A , O, R )) Xavier HINAUT - Inria, Bordeaux, France 18

Input: structure of sentence « The W X -s the Y to the Z » 1 st 2 nd Output: meaning (predicate) Give ( Boy , Ball , Dog ) 4 th 3 rd « The boy give –s the ball to the dog » [Sem. W. ] = W, X, Y, Z By That -s To The [Sem. W] Hinaut & Dominey 2013 Predicate Agent Object Recipient 1 st Sem. Word (W) Predicate Agent Object Recipient 2 nd Sem. Word (X) Predicate Agent Object Recipient 3 rd Sem. Word (Y) Predicate Agent Object Recipient 4 th Sem. Word (Z) Xavier HINAUT - Inria, Bordeaux, France 19

After training … Testing Hinaut & Dominey 2013 By That Predicate Agent Objet Recipient … Sem. Word Predicate Agent Objet Recipient 3 rd Sem. Word -s To The [Sem. W] « The cat was fed by the girl » Xavier HINAUT - Inria, Bordeaux, France 20

More complex sentences Hinaut & Dominey 2013 Xavier HINAUT - Mnémosyne 21

More complex sentences Hinaut & Dominey 2013 Xavier HINAUT - Mnémosyne 22

More complex sentences Hinaut & Dominey 2013 Xavier HINAUT - Mnémosyne 23

More complex sentences Hinaut & Dominey 2013 Xavier HINAUT - Mnémosyne 24

More complex sentences Hinaut & Dominey 2013 Xavier HINAUT - Mnémosyne 25

More complex sentences Hinaut & Dominey 2013 Adds interpretability to the outputs of the model Can be used to detect problems in sentences: e. g. Speech recognition errors creates “bump” in the outputs Can be used as a confidence measure Xavier HINAUT - Mnémosyne 26

Generation of training corpora Belle Marquise, vos beaux yeux me font mourir d’amour. D’amourir me font, belle Marquise, vos beaux yeux. Vos yeux beaux d’amour me font, belle Marquise, mourir. Mourir vos beaux yeux, belle Marquise, d’amour me font. Me font vos yeux beaux mourir, belle Marquise, d’amour. • Le Bourgeois Gentilhomme. Molière. Xavier HINAUT - Mnémosyne 27

Generation of training corpora Belle Marquise, vos beaux yeux me font mourir d’amour, D’amourir me font, belle Marquise, vos beaux yeux, Vos yeux beaux d’amour me font, belle Marquise, mourir, Mourir vos beaux yeux, belle Marquise, d’amour me font, Me font vos yeux beaux mourir, belle Marquise, d’amour, • Le Bourgeois Gentilhomme. Molière. Xavier HINAUT - Mnémosyne 28

Small Corpus (462) • 1 main (m) and 1 relative (r) clauses • 6 semantic words at max. per construction • All possible semantic word (role) orders 462 constructions – – – – Some non natural word order Roles order for main and relative cl. : “ex. of sentence” m(AP), r(): "The giraffe walk -s. " m(AOP), r(): "By the beaver the fish was cut -ed. " m(APRO), r(): "The dog give -s to the mouse the cat. " m(APO), r(AP): "The beaver that think -s cut -s the fish. " m(PA), r(AP): "Walk -ing was the giraffe that think -s. " m(PARO), r(OAP): "Give -ed by the dog to the mouse that the guy kiss s was the ball. " Hinaut & Dominey 2013 Xavier HINAUT - Mnémosyne 29

Batch learning (for 462 corpus) Hinaut & Dominey 2013 Xavier HINAUT - Mnémosyne 30

Big Corpus (90, 000) Hinaut & Dominey 2013 Xavier HINAUT - Mnémosyne 31

Generalization on big corpus (90 k) Hinaut & Dominey 2013 Xavier HINAUT - Mnémosyne 32

Generalization on big corpus (90 k) Hinaut & Dominey 2013 Xavier HINAUT - Mnémosyne 33

Fully online training version with LMS Win Xavier HINAUT - Mnémosyne W Wout 34

Fully online training version with LMS Win W Wout Weights are modified only at the end of the sentence (During the last 2 time steps) Xavier HINAUT - Mnémosyne 35

The dog was scratch –ed by the cat that see –s the mouse. Output activity dog scratch cat Time Xavier HINAUT - Mnémosyne 36

Xavier HINAUT - Inria, Bordeaux, France 37

Towards Modelling Language Acquisition Language acquisition Language Brain Model Grounded Language Tomasello 2003 Xavier HINAUT - Mnémosyne 38

Sentence Comprehension Hinaut et al. 2014 Xavier HINAUT - Mnémosyne 39

• Video links: – i. Cub sentence comprehension • https: //youtu. be/AUb. JAupk. U 4 M – Nao « Humanoidly Speaking » video for IJCAI 2015 • https: //youtu. be/R 4 c. E 4 b. Ah. Lr. U Xavier HINAUT - Mnémosyne 40

Hinaut et al. 2014 41

Sentence Production Hinaut et al. 2014 Xavier HINAUT - Inria, Bordeaux, France 42

Sentence Production • Video link: – i. Cub sentence production • https: //youtu. be/3 Ze. PCuvygi 0 Xavier HINAUT - Inria, Bordeaux, France 43

Some sentences said by users • • • point the triangle before grasping the circle put the cross to the left before grasping the circle point to the cross twice before you grasp the cross please grasp the triangle before pushing the triangle to the middle please push the cross to the right grasp the circle and then point to it touch the triangle then move it to the left the cross touch it point to the circle after having grasped it Hinaut et al. 2014 Xavier HINAUT - Mnémosyne 44

Summary (HRI) Xavier HINAUT - Mnémosyne 45

Generalizing on French & English FR EN Hinaut et al, . 2015 Xavier HINAUT - Mnémosyne 46

Training data set example 1 st command; 2 nd command Do this then do that Point(triangle); Grasp(circle) point the triangle before grasping the circle Grasp(triangle); Grasp(cross) before you grasp the cross please grasp the triangle Point(cross); Point(cross) point to the cross twice Touch(triangle); Move(triangle, left) touch the triangle then move it to the left Hinaut et al. 2014

Meaning slots representation This is a sentence to be understood • check if • if husband home • husband my • if five Check if my husband is at home at five • chequea si • si marido casa • marido mi • si cinco Chequea si mi marido està en casa a las cinco • überprüfe ob • ob Mann Hause • Mann mein • ob fünf Überprüfe, ob mein Mann um fünf zu Hause ist • vérifie si • si mari maison • mari mon • si heures cinq Vérifie si mon mari est à la maison à cinq heures Xavier HINAUT - Inria, Bordeaux, France Hinaut & Twiefel (in revision) 48

Proof-of-concept for 15 languages (with different word order) 1) answer phone ; answer the phone 2) geh an Telefon ; geh an das Telefon 3) contesta teléfono ; contesta el teléfono 4) réponds téléphone ; réponds au téléphone 5) rispondi telefono ; rispondi al telefono 6) contesta telèfon ; contesta el telèfon 7) 接电话; 接那-通电话 8) 接電話; 接那-通電話 9) jawab telefon ; jawab -kan panggilan telefon itu 10) cevap ver telefon ; telefon -a cevap ver 11) uchal phone ; phone uchal 12) dho javaab phon ; phon ka javaab dho 13) telefono erantzun ; telefono -a erantzun 14) javab telfon ; telfon ra javab bede 15) atenda telefone ; atenda o telefone 16) вдигни телефон ; вдигни телефон -a Hinaut & Twiefel (in revision) 1) English 2) German 3) Spanish 4) French 5) Italian 6) Catalan 7) Simpl. Mandarin 8) Trad. Mandarin 9) Malay 10) Turkish 11) Marathi 12) Hindi 13) Basque 14) Persian 15) Portuguese 16) Bulgarian Xavier HINAUT – Inria, Bordeaux, France – Mnemosyne Team

Is speech recognition solved (with Deep Learning)? • Using cloud speech API (Google) • Trying to recognize robot commands – Output the N-best sentence results • “Point to the circle” – "going to the circus", "point to the suckers", "point to the circle", "went to the circus” • “Grasp the cross” – "brass knuckles", "grasp the cross", "cross the cross", "crust bros", "trust the cross", "craft bros", "frost bros", "crous bros", "cross bros", "crisp bros", "christ bros", "chris bros", "cauterets", "crust brothers", "recette bros", "crust triche", "cremi bros” • “Touch the circle then point to the circle” – "that's the circle then point to the circle", "that's the second then point to the circle", "fastest organ 10. 2 ii", "that's the second point to the circle", "that's the circle 10. 2 ii” – Not found Xavier HINAUT - Mnémosyne 50

Is speech recognition solved (with Deep Learning)? • “Touch the triangle and put it on the left” – "does the triangle and put it on the list", "that's the triangle and put it on the left", "what's the triangle and put it on the list", "tesco triangle and put it on the list", "danse the triangle and put it on the list", "is the triangle and put it on the list", "that's the triangle and put it on the lift", "that the triangle and put it on the list", "that's a triangle and put it on the list", "what is the triangle and put it on the list", "it's the triangle and put it on the list", "that's the langon and put it on the list", "that's the triangle and put it on the laas” – Not found • What can we do to solve this problem? Xavier HINAUT - Mnémosyne 51

From Words towards Phonemes Xavier HINAUT – Inria, Bordeaux, France – Mnemosyne Team

Works with seq. of words … Hinaut 2018 Xavier HINAUT – Inria, Bordeaux, France – Mnemosyne Team

… and with seq. of phonemes! AO 1 N DH AH 0 P UH T … Hinaut 2018 Xavier HINAUT – Inria, Bordeaux, France – Mnemosyne Team

From Words towards Phonemes Hinaut 2018 Xavier HINAUT – Inria, Bordeaux, France – Mnemosyne Team

Works with seq. of phonemes! Hinaut 2018 Xavier HINAUT – Inria, Bordeaux, France – Mnemosyne Team

Works with seq. of phonemes! Hinaut 2018 Xavier HINAUT – Inria, Bordeaux, France – Mnemosyne Team

Works with seq. of phonemes! Hinaut 2018 Xavier HINAUT – Inria, Bordeaux, France – Mnemosyne Team

Works with seq. of phonemes! Hinaut 2018 Xavier HINAUT – Inria, Bordeaux, France – Mnemosyne Team

Works with seq. of phonemes! Hinaut 2018 Xavier HINAUT – Inria, Bordeaux, France – Mnemosyne Team

Works with seq. of phonemes! Hinaut 2018 Xavier HINAUT – Inria, Bordeaux, France – Mnemosyne Team

Summary on Neural Parsing Model • Generic neural circuit for sequence processing • Online and anytime (can be probed before the end of the sentence) • Generalize – Corpus size from 20 to 90, 000 – Even with “Out-of-Vocabulary” (unknown) words – On seq. of phonemes, seq. of words & grammatical constructions • Enable robots to learn non-stereotypical sentences • Same circuit instance can generalize from different inputs – French & English • Proof of concept for 15 Asian-European languages • Flexible outputs – no need to have predicates Xavier HINAUT - Inria, Bordeaux, France 62

On-going & Future work • Enhance this sentence parsing model – Apply “phoneme” version with audio recordings • Deep. Speech outputing phonemes / words • Show better global perf. (speech recog. + understanding) • Crowdsourcing website – Collect data in several language • Speech / written sentences • Meanings (predicates or any representation) • Users correct each other (quality of data, . . . ) – Robot simulator provide actions to be described – YOU can provideos to be annotated! Xavier HINAUT - Mnémosyne 63

Collaborators • Lyon P. F. Dominey M. Petit … • Hamburg S. Wermter J. Twiefel L. Mici S. Magg … • Paris C. del Negro A. Cazala. . . • Bordeaux S. Pagliarini A. Strock F. Alexandre N. Rougier A. Leblois … github. com/neuronal. X Questions? 64

References • • • Hinaut X, Dominey PF (2011) A three-layered model of primate prefrontal cortex encodes identity and abstract categorical structure of behavioral sequences. J Physiol - Paris 105: 16– 24. Hinaut, X, Dominey PF (2012) On-Line Processing of Grammatical Structure Using Reservoir Computing. In Artificial Neural Networks and Machine Learning – ICANN 2012, LNCS vol. 7552, 2012, pp 596 -603. Hinaut, X, Dominey PF (2013) Real-Time Parallel Processing of Grammatical Structure in the Fronto-Striatal System: A Recurrent Network Simulation Study Using Reservoir Computing. Plo. S ONE 8(2): e 52946. Hinaut X, Petit M, Pointeau G Dominey PF (2014) Exploring the Acquisition and Production of Grammatical Constructions Through Human-Robot Interaction with Echo State Networks. Front. Neurorobot. 8: 16. Hinaut X, Wermter, S (2014) An Incremental Approach to Language Acquisition: Thematic Role Assignment with Echo State Networks. ICANN. Xavier HINAUT - Mnémosyne 65

References • • Hinaut, X. , Lance, F. , Droin, C. , Petit, M. , Pointeau, G. , & Dominey, P. F. (2015) Corticostriatal response selection in sentence production: Insights from neural network simulation with reservoir computing. Brain and language, 150, 54 -68. Hinaut, X. , Twiefel, J. , Soares, M. B. , Barros, P. , Mici, L. , & Wermter, S. (2015). Humanoidly speaking–learning about the world and language with a humanoid friendly robot. In International Joint Conference on Artificial Intelligence Video competition. https: //youtu. be/R 4 c. E 4 b. Ah. Lr. U X. Hinaut, J. Twiefel, M. Petit, P. F. Dominey, and S. Wermter (2015) “A recurrent neural network for multiple language acquisition: Starting with english and french, ” in NIPS 2015 Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches. Mici, L. , Hinaut, X. , & Wermter, S. (2016) Activity recognition with echo state networks using 3 D body joints and objects category. In European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN) (pp. 465 -470). Xavier HINAUT - Mnémosyne 66

References • • Twiefel, J. , Hinaut, X. , Borghetti, M. , Strahl, E. , & Wermter, S. (2016) Using natural language feedback in a neuro-inspired integrated multimodal robotic architecture. In Robot and Human Interactive Communication (RO-MAN), 2016 25 th IEEE International Symposium on (pp. 52 -57). Hinaut, X. , Twiefel, J. , & Wermter, S. (2016) Recurrent neural network for syntax learning with flexible predicates for robotic architectures. In Development and Learning and Epigenetic Robotics (ICDL-Epi. Rob), 2016 Joint IEEE International Conference on (pp. 150 -151) Twiefel, J. , Hinaut, X. , & Wermter, S. (2016) Semantic role labelling for robot instructions using echo state networks. In European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN). Twiefel, J. , Hinaut, X. , & Wermter, S. (2017) Syntactic Reanalysis in Language Models for Speech Recognition. In 2017 Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-Epi. Rob). Xavier HINAUT - Mnémosyne 67

References • • Hinaut, X. , & Twiefel, J. (2017). Teach Your Robot Your Language! Trainable Neural Parser for Modelling Human Sentence Processing: Examples for 15 Languages. (preprint HAL) Strock, A. , Rougier, N. , & Hinaut, X. (2018). A Simple Reservoir Model of Working Memory with Real Values. Prooc. of IJCNN, Rio de Janeiro, Brasil, July 2018. – Code: https: //github. com/anthony-strock/ijcnn 2018 • Hinaut, X. (2018). Which Input Abstraction is Better for a Robot Syntax Acquisition Model? Phonemes, Words or Grammatical Constructions? In 2018 Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-Epi. Rob). – Supp. Mat. & Code: https: //github. com/neuronal. X/Hinaut 2018_icdl-epirob • Pagliarini, S. , Hinaut, X. , Leblois, A. (2018) A Bio-Inspired Model towards Vocal Gesture Learning in Songbird. In 2018 Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-Epi. Rob). Xavier HINAUT - Mnémosyne 68