WP 5 Cognitive Integration and Bootstrapping Angelo Cangelosi
WP 5 Cognitive Integration and Bootstrapping Angelo Cangelosi, Alessandra Sciutti, Metta, Sagerer, Nolfi, Nehaniv, Fischer , Tani, Belpaeme, Fadiga, Nori, Rohlfing, Wrede, Tuci, Tikhanoff, Schillingmann, Lohan, Mirolli, Saunders, Dautenhahn, Zeschel, Marocco, Peniak, Foerster, Massera, Sato, Lyon, Pitsch, Gijsberts, Beck, et al.
Overview 1. WP 5 Overview: The integration WP? 2. Task 5. 1 (Roadmap) • • Road Map preparation Results and dissemination 3. Task 5. 3 (HRI and neuroscience) • • Planned experiments Main result(s) 4. Task 5. 2 (language as cognitive tool) • Planned experiments 5. Integration Outlook ITALK Year 1 Review Düsseldorf, 1 July 2009
Overview 1. WP 5 Overview: The integration WP? 2. Task 5. 1 (Roadmap) • • Road Map preparation Results and dissemination 3. Task 5. 3 (HRI and neuroscience) • • Planned experiments Main result(s) 4. Task 5. 2 (language as cognitive tool) • Planned experiments 5. Integration Outlook ITALK Year 1 Review Düsseldorf, 1 July 2009
The Integration WP? • Different levels of integration of work 1. Intra-WP: Each WP includes integration of cognitive capabilities • E. g. WP 1 action with language (T 1. 4 T 4. 3) • E. g. WP 3 social with language (T 3. 1) 2. Inter-WP: WP 5 for additional aspects of cognitive integration (bootstrapping) • Roadmap vision (T 5. 1) • Extra-linguistic roles of language (T 5. 2) • From neuroscience to HRI (T 5. 3, T 5. 4) 3. Work (eg. results, software) integration • Standardisation process (i. Cub/ITALK repository)
WP 5 Objectives • WP Objectives (Year 1) – to identify the underlying methodological, theoretical and technological principles for research on the ROADMAP integration of linguistic, motor and cognitive capabilities in robots; – to uncover, from both a theoretical and an application point of view, the relationships between the different levels of integration and transfer of action and language knowledge through the involvement of the mirror. Gaze experiments neuron system and the role of motor affordances in human-robot communication experiments – the understand the role language as atool cognitive tool Language asof cognitive and its role in the bootstrapping of cognitive capabilities such as attention and gaze sharing, syntax processing, and object representation through motor affordances
Overview 1. WP 5 Overview: The integration WP? 2. Task 5. 1 (Roadmap) • • Road Map preparation Results and dissemination 3. Task 5. 3 (HRI and neuroscience) • • Planned experiments Main result(s) 4. Task 5. 2 (language as cognitive tool) • Planned experiments 5. Integration Outlook ITALK Year 1 Review Düsseldorf, 1 July 2009
ROADMAP: Process • Meeting 1: Planning of process • Meeting 2: Rome Workshop • Drafting – 5 Thematic groups – Email co-writing • Dissemination – Website – Submission to IEEE TAMD journal
ROADMAP: Intl. Workshop • Participants – All ITALK principal scientists – ISAB/Advisors: • • I. Pepperberg (Harvard USA) M. Bowermann (Max Planck) A. Glenberg (Arizona State) R. Kubat (MIT USA, Roy’s lab) K. Plunkett (Oxford UK) S. Harnad (UQAM & Southampton University. ) R. Grush (UCSD USA) – Related EU projects: • L. Steels (ALEAR), T. Ziemke (ICEA), A. Borghi (ROSSI), P. De Ruiter (JAST), P. Dominey (CHRIS)
ROADMAP: Intl. Workshop
ROADMAP: Post-Workshop • Thematic Groups I. Introduction/Vision/Milestones: led by A. Cangelosi (PLYM) with contribution from T. Belpaeme and all other staff II. Action Learning Challenges: led by G. Metta and L. Fadiga (IIT) with contribution from PLYM staff III. Category Learning Challenges: led by S. Nolfi (CNR) with contribution from PLYM and RIKEN staff IV. Social Learning Challenges: led by G. Sagerer (BIEL) with contribution from UH and USD staff V. Language Learning Challenges: led by K. Fischer (USD) with contribution from BIEL, PLYM and RIKEN staff
ROADMAP Results: Key Challenges I. Understanding how agents learn and represent compositional and hierarchical actions; II. Understanding how agents acquire and represent compositional lexicons and linguistic knowledge; III. Understanding the dynamics of social interaction and social learning, IV. Understanding of how compositional action and language representations are integrated to bootstrap cognition.
Cognitive integration Social learning Language learning Action learning ROADMAP Results: Milestones Developmental learning of simple actions (primitives) Capacity to categorise and name objects, events and states Acquisition of hierarchical and compositional actions Learning the association between syntactic constructions and composite actions via social learning Social based acquisition of action generalization rules Ability to correlate action and language generalization capabilities Acquisition of the ability to generalize over goals Ability to correlate recursive /composite actions with recursive linguistic expressions Ability to learn rich action repertoires based on social/linguistic descriptions Naming of individual objects Acquisition of early, holophrastic, natural language utterances in embodied learning tasks Naming of individual events and states Embodied learning of itemspecific early syntactic constructions from verbal human-robot interactions Bootstrapping of linguistic capabilities from embodied verbal interaction with the acquisition of compositional language structures Acquisition and deployment of contextually embedded syntactic constructions through social interaction Development of generalpurpose grammatical constructions and syntactic competence Scalable lexicon, grammar and discourse learning from embodied linguistic interaction with humans Interactional acquisition of symbolic communication systems sharing relevant properties with natural language Harnessing of elementary non-verbal social cues (gaze, turn-taking, mirroring etc. ) to enhance social learning for language and skill acquisition Modelling intermodal learning (acoustic packaging) Development of a tutor spotter for social learning scenarios Joint intentional framing and referential intent Acquisition of negation usage of various types (e. g. refusal, absence, prohibition, propositional denial) Development of architectures capable of exploiting pragmatic skills such as sequential interactional organization (contingency, turn-taking) and use of prosody for grammatical learning Harnessing of Model/Rival (M/R) learning, motivational systems and predictive social interaction Exploiting interactions of prosody, internal motivation, inter-subjectivity and pragmatics in language acquisition and dialogue Developing architectures based on intermodal learning and sensitivity to a tutor Temporally extended understanding of the social motivations and intentions of other minds, context, and (auto)biographic and narrative (re)construction. Development of first systems that are capable of social learning and sequential organisation of interaction in specific scenarios Development of systems that are capable of social learning and pragmatic organisation of interaction in various scenarios Integration of basic action and naming representations and emergence of shared representation roles for both actions and names Simulation of Action- Computational neuroscience Language Compatibility models of action and effects. language integration Co-evolution of action and language skills for simple grounded lexicons. Use of general purpose grammatical constructions for the creation of new complex motor and perceptual concepts Scalable lexicons of abstract concepts based on the developmental acquisition of a grounding kernel Acquisition of open repertoires of compositional actions and lexicons sharing natural language properties Next 2 Years Next 4 Years Next 6 -8 Years TIME Next 10 Years Next 15 Years Next 20 Years
Cognitive integration Social learning Language learning Action learning ROADMAP Results: Milestones Developmental learning of simple actions (primitives) Capacity to categorise and name objects, events and states Acquisition of hierarchical and compositional actions Learning the association between syntactic constructions and composite actions via social learning Social based acquisition of action generalization rules Ability to correlate action and language generalization capabilities Acquisition of the ability to generalize over goals Ability to correlate recursive /composite actions with recursive linguistic expressions Ability to learn rich action repertoires based on social/linguistic descriptions Naming of individual objects Acquisition of early, holophrastic, natural language utterances in embodied learning tasks Naming of individual events and states Embodied learning of itemspecific early syntactic constructions from verbal human-robot interactions Bootstrapping of linguistic capabilities from embodied verbal interaction with the acquisition of compositional language structures Acquisition and deployment of contextually embedded syntactic constructions through social interaction Development of generalpurpose grammatical constructions and syntactic competence Scalable lexicon, grammar and discourse learning from embodied linguistic interaction with humans Interactional acquisition of symbolic communication systems sharing relevant properties with natural language Harnessing of elementary non-verbal social cues (gaze, turn-taking, mirroring etc. ) to enhance social learning for language and skill acquisition Modelling intermodal learning (acoustic packaging) Development of a tutor spotter for social learning scenarios Joint intentional framing and referential intent Acquisition of negation usage of various types (e. g. refusal, absence, prohibition, propositional denial) Development of architectures capable of exploiting pragmatic skills such as sequential interactional organization (contingency, turn-taking) and use of prosody for grammatical learning Harnessing of Model/Rival (M/R) learning, motivational systems and predictive social interaction Exploiting interactions of prosody, internal motivation, inter-subjectivity and pragmatics in language acquisition and dialogue Developing architectures based on intermodal learning and sensitivity to a tutor Temporally extended understanding of the social motivations and intentions of other minds, context, and (auto)biographic and narrative (re)construction. Development of first systems that are capable of social learning and sequential organisation of interaction in specific scenarios Development of systems that are capable of social learning and pragmatic organisation of interaction in various scenarios Integration of basic action and naming representations and emergence of shared representation roles for both actions and names Simulation of Action- Computational neuroscience Language Compatibility models of action and effects. language integration Co-evolution of action and language skills for simple grounded lexicons. Use of general purpose grammatical constructions for the creation of new complex motor and perceptual concepts Scalable lexicons of abstract concepts based on the developmental acquisition of a grounding kernel Acquisition of open repertoires of compositional actions and lexicons sharing natural language properties Next 2 Years Next 4 Years Next 6 -8 Years TIME Next 10 Years Next 15 Years Next 20 Years
ROADMAP: Publication • Roadmap white paper (website) • Publication (submitted) • Workshop proceedings (CDROM and on-line)
Overview 1. WP 5 Overview: The integration WP? 2. Task 5. 1 (Roadmap) • • Road Map preparation Results and dissemination 3. Task 5. 3 (HRI and neuroscience) • • Planned experiments Main result(s) 4. Task 5. 2 (language as cognitive tool) • Planned experiments 5. Integration Outlook ITALK Year 1 Review Düsseldorf, 1 July 2009
Task 5. 3: goals • Human-robot experiments and mirror-neuron system: Gaze behaviour during observation of actions performed by a humanoid artefact – Human-human interaction: mo 6 -24 – Human-robot interaction: mo 24 -30 ITALK Year 1 Review Düsseldorf, 1 July 2009
Proactive gaze behaviour and mirror neurons • Gaze behaviour is proactive (Flanagan and Johansson 2003) fundamental in interactions (also human – robot). • Which are the characteristics of the observed action that evoke the pro-active gaze behaviour? Eye movements are used to reflect the activity of a motor resonance involving a mirror-neuron system. • Proactive gaze behavior as a function of: – Observed action kinematics – Cognitive cues ITALK Year 1 Review Düsseldorf, 1 July 2009
Experiments Effects of the observation of lifting objects of different weight on: • Mirror neurons involvement – Previous experiment • Proactive gaze behaviour – New ongoing pilot experiment ITALK Year 1 Review Düsseldorf, 1 July 2009
Object lifting task Optotrak cameras Optotrak markers Eyelink system ITALK Year 1 Review Düsseldorf, 1 July 2009
Stimuli Visible heavy vs. Visible light ITALK Year 1 Review Object appearance Demonstrator’s kinematics Hidden heavy vs. Hidden light Demonstrator’s kinematics only Labeled heavy vs. Labeled light (both heavy) Object appearance only: cognitive cues Düsseldorf, 1 July 2009
EMG : FDI TMS : 120% of MT 50 ms after lifting Normalized MEP area (Z-score) Mirror neurons involvement Protocol: 8 subjects, 10 trials, Conditions randomized ITALK Year 1 Review Düsseldorf, 1 July 2009
Hand – Gaze Measures ITALK Year 1 Review Düsseldorf, 1 July 2009
Preliminary results HAND & GAZE HAND Kinematics Lifting ITALK Year 1 Review Placing Light Heavy Observer Demonstrator Düsseldorf, 1 July 2009
Current and future steps • Currently: setup ready, starting data collection. • Next steps: – Modulating predictability of demonstrator’s motion (e. g. alternating different arrival points) – Robot as demonstrator • Modulation of kinematics • Modulation of interaction ITALK Year 1 Review Düsseldorf, 1 July 2009
Overview 1. WP 5 Overview: The integration WP? 2. Task 5. 1 (Roadmap) • • Road Map preparation Results and dissemination 3. Task 5. 3 (HRI and neuroscience) • • Planned experiments Main result(s) 4. Task 5. 2 (language as cognitive tool) • Planned experiments 5. Integration Outlook ITALK Year 1 Review Düsseldorf, 1 July 2009
T 5. 2: Language as a cognitive tool and its role in enhancing cognition (e. g. inner speech) The aim of this task is that to verify, through robotic experiments, whether: • the co-development of behavioral and linguistic skills represent a crucial prerequisite for the development of complex cognitive skills • linguistically mediated social interactions cause a radical transformation of elementary cognitive abilities, as proposed by Vygotsky (1962, 1978) • Talking-to-oneself (inner speech) as a cognitive aid tool (Vygotsky 1978)
Objects manipulation scenario A robot located in front of few objects will be trained (through a trial and error process) for displaying: (1) few behavioural skills (e. g. “touching the blue object”, “indicating the red object”, or “moving the green object”) (2) Associated linguistic skills that will consists in “verbally describing the behaviour that the robot is currently exhibiting” and/or “listening to a verbal command executing the corresponding behaviour”
Shape recognition/drawing scenario A robot located in front of a white board will be trained (through a trial and error process) for: (1)Drawing a shapes (e. g. simple geometrical shapes such us a square or a circle or more elaborate shapes) by moving a pen after having observed the experimenter drawing the same shape (2)Verbally describing the shape being drawn by the experimenter or by the robot itself
Research Objectives Through these experiments we will investigate: (1) Whether the possibility to be exposed to linguistic inputs might facilitate the development of the required behavioural and cognitive skills. (2) Whether the need to produce linguistic outputs can facilitate the development of the required behavioural and cognitive skills (3) Whether the possibility of the agent to access own linguistic outputs (inner speech) can represent a prerequisite for the development of certain behavioral and cognitive skills (4) How the internal categories developed by the agents are co-shaped by the development of behavioral and linguistic skills
Overview 1. WP 5 Overview: The integration WP? 2. Task 5. 1 (Roadmap) • • Road Map preparation Results and dissemination 3. Task 5. 3 (HRI and neuroscience) • • Planned experiments Main result(s) 4. Task 5. 2 (language as cognitive tool) • Planned experiments 5. Integration Outlook ITALK Year 1 Review Düsseldorf, 1 July 2009
WP 5 Outlook: Integration for Cognitive Bootstrapping • Additional dimensions in cognitive integration – E. g. Neuroscience and mirror neuron system (Tasks 5. 3 -5. 4) Mirror neuron involvement on gaze in humans Mirror neuron involvement on gaze in HRI Mirror neuron involvement on linguistic complexity in HRI – E. g. Self-talk (Tasks 5. 2) Language for communication Language for taking-to-oneself Language as cognitive aid tool language/cognition relativism
- Slides: 31