fa buf cse lo The SNe PS Approach

fa buf @ cse lo The SNe. PS Approach to Cognitive Robotics Stuart C. Shapiro Department of Computer Science and Engineering and Center for Cognitive Science University at Buffalo shapiro@cse. buffalo. edu S. C. Shapiro

fa buf @ cse • • lo Outline Introduction Intensional Representation & Propositions SNe. PS Connectives and Quantifiers SNe. RE Acting Constructs Example Plans Representation and Use of Indexicals A Personal Sense of Time Summary S. C. Shapiro

fa buf @ cse lo Goal • A computational cognitive agent that can: – Understand communicate in English; – Discuss specific, generic, and “rule-like” information; – Reason; – Discuss acts and plans; – Sense; – Act; – Remember and report what it has sensed and done. S. C. Shapiro

fa buf @ cse lo Embodied Cassie • A computational cognitive agent – Embodied in hardware – or Software-Simulated – Based on SNe. PS and GLAIR. S. C. Shapiro

fa buf @ cse lo SNe. PS • Knowledge Representation and Reasoning – Propositions as Terms • SNIP: SNe. PS Inference Package – Specialized connectives and quantifiers • SNe. BR: SNe. PS Belief Revision • SNe. RE: SNe. PS Rational Engine • Interface Languages – SNe. PSUL: Lisp-Like – SNe. PSLOG: Logic-Like – GATN for Fragments of English. S. C. Shapiro

fa buf @ cse lo GLAIR Architecture Grounded Layered Architecture with Integrated Reasoning Knowledge Level NL SNe. PS Perceptuo-Motor Level Sensory-Actuator Level Vision Sonar Motion Proprioception S. C. Shapiro

fa buf @ cse lo Interaction with Cassie English (Statement, Question, Command) GATN (New Belief) [SNe. PS] Answer [SNIP] GATN (Current) Set of Beliefs [SNe. PS] Reasoning Parser Clarification Dialogue Looking in World Actions (Updated) Set of Beliefs [SNe. RE] [SNe. PS] Generator Reasoning English sentence expressing new belief answering question reporting actions S. C. Shapiro

fa buf @ cse lo Cassie, the Blocks. World Robot S. C. Shapiro

fa buf @ cse S. C. Shapiro lo Cassie, the FEVAHR

fa buf @ cse S. C. Shapiro lo FEVAHR/Cassie in the Lab

fa buf @ cse S. C. Shapiro lo FEVAHRWorld Simulation

fa buf @ cse lo UXO Remediation Cassie Corner flag Field UXO Drop-off zone Non. UXO object Battery meter Corner flag Recharging Station S. C. Shapiro Corner flag Cassie Safe zone

fa buf @ cse S. C. Shapiro lo Crystal Space Environment

fa buf @ cse • • lo Outline Introduction Intensional Representation & Propositions SNe. PS Connectives and Quantifiers SNe. RE Acting Constructs Example Plans Representation and Use of Indexicals A Personal Sense of Time Summary S. C. Shapiro

fa buf @ cse lo Entities, Terms, Symbols, Objects • Cassie’s mental entity: a person named Bill • SNe. PS term: B 5 • Object in world: S. C. Shapiro

fa buf @ cse lo Intensional Representation Intensional entities are distinct even if coreferential. “The morning star is the evening star. ” “George IV wondered if Scott was the author of Waverly. ” S. C. Shapiro

fa buf @ cse lo Mc. Carthy’s Telephone Number Problem Mary's telephone number is Mike's telephone number. I understand that Mike's telephone number is Mary's telephone number. Pat knew Mike's telephone number. I understand that Pat knew Mike's telephone number. Pat dialed Mike's telephone number. I understand that Pat dialed Mike's telephone number. S. C. Shapiro

fa buf @ cse lo Answering the Telephone Number Problem Did Pat dial Mary's telephone number? Yes, Pat dialed Mary's telephone number. Did Pat know Mary's telephone number? I don't know. S. C. Shapiro

fa buf @ cse lo Representing Propositions must be first-class entities of the domain Represented by terms. S. C. Shapiro

fa buf @ cse lo Discussing Propositions That Bill is sweet is Mary's favorite proposition. I understand that Mary's favorite proposition is that Bill is sweet. Mike believes Mary's favorite proposition. I understand that Mike believes that Bill is sweet. S. C. Shapiro

fa buf @ cse • • lo Outline Introduction Intensional Representation & Propositions SNe. PS Connectives and Quantifiers SNe. RE Acting Constructs Example Plans Representation and Use of Indexicals A Personal Sense of Time Summary S. C. Shapiro

fa buf @ cse lo Logic for NLU & Commonsense Reasoning Either Pat is a man or Pat is a woman or Pat is a robot. I understand that Pat is a robot or Pat is a woman or Pat is a man. Pat is a woman. I understand that Pat is a woman. What is Pat? Pat is a woman and Pat is not a robot and Pat is not a man. S. C. Shapiro

fa buf @ cse lo Representation in FOPL? Man(Pat) Woman(Pat) Robot(Pat) S. C. Shapiro

fa buf @ cse lo Representation in FOPL? Man(Pat) Woman(Pat) Robot(Pat) but don’t want inclusive or S. C. Shapiro

fa buf @ cse lo Representation in FOPL? Man(Pat) Woman(Pat) Robot(Pat) but don’t want inclusive or Man(Pat) + Woman(Pat) + Robot(Pat) T T T F T So don’t want exclusive or either S. C. Shapiro

fa buf @ cse lo andor(i, j){P 1, . . . , Pn} True iff at least i, and at most j of the Pi are True S. C. Shapiro

fa buf @ cse lo thresh(i, j){P 1, . . . , Pn} True iff either fewer than i, or more than j of the Pi are True Note: thresh(i, j) ~andor(i, j) S. C. Shapiro

fa buf @ cse lo or-entailment {P 1, . . . , Pn} v=> {Q 1, . . . , Qn} True iff S. C. Shapiro for all i, j Pi Qj

fa buf @ cse lo and-entailment {P 1, . . . , Pn} &=> {Q 1, . . . , Qn} True iff for all j P 1 &…& Pn Qj S. C. Shapiro

fa buf @ cse lo Numerical entailment {P 1, . . . , Pn} i=> {Q 1, . . . , Qn} True iff for all j andor(i, n){P 1, …, Pn } Qj S. C. Shapiro

fa buf @ cse lo Universal Quantifier all(ū)({R 1(ū), . . . , Rn(ū)} &=> {C 1(ū), . . . , Cm(ū)}) Every ā that satisfies R 1(ū)&…& Rn(ū) also satisfies C 1(ū), . . . , Cm(ū)}) S. C. Shapiro

fa buf @ cse lo Numerical Quantifiers nexists(i, j, k)(x) ({P 1(x), . . . , Pn(x)}: {Q(x)})} There are k individuals that satisfy P 1(x) . . . Pn(x) and, of them, at least i and at most j also satisfy Q(x) S. C. Shapiro

fa buf @ cse • • lo Outline Introduction Intensional Representation & Propositions SNe. PS Connectives and Quantifiers SNe. RE Acting Constructs Example Plans Representation and Use of Indexicals A Personal Sense of Time Summary S. C. Shapiro

fa buf @ cse lo MENTAL ACTS • Believe(proposition) • Disbelieve(proposition) S. C. Shapiro

fa buf @ cse lo Act Selection • Do-One({act 1. . . actn}) • Snif(if(condition, act), . . . if(condition, act) [else(act)]) S. C. Shapiro

fa buf @ cse lo Act Iteration • Do-All({act 1. . . actn}) • Sniterate(if(condition, act), . . . if(condition, act), [else(act)]) • Snsequence(act 1, . . . , actn) • Cascade(act 1, . . . , actn) • P-Do-All({act 1, . . . , act 2}) S. C. Shapiro

fa buf @ cse lo Entity Iteration With. Some(var, suchthat, do, [else]) With. All(var, suchthat, do, [else]) With. Some+(var, suchthat, do, [else]) With. New(vars, thatare, suchthat, do, [else]) S. C. Shapiro

fa buf @ cse lo Proposition/Act Transformers • • Achieve(proposition) Act. Plan(act, plan) Goal. Plan(proposition, act) Precondition(act, proposition) Effect(act, proposition) When. Do(proposition, act) Whenever. Do(proposition, act) If. Do(proposition, act) S. C. Shapiro

fa buf @ cse • • lo Outline Introduction Intensional Representation & Propositions SNe. PS Connectives and Quantifiers SNe. RE Acting Constructs Example Plans Representation and Use of Indexicals A Personal Sense of Time Summary S. C. Shapiro

fa buf @ cse lo Conditional Plans If a block is on a support then a plan to achieve that the support is clear is to pick up the block and then put the block on the table. all(x, y) ({Block(x), Support(y), On(x, y)} &=> {Goal. Plan(Clear(y), Snsequence(Pickup(x), Put(x, Table)))}) STRIPS-like representation: No times S. C. Shapiro

fa buf @ cse lo Use of Conditional Plan Goal. Plan(Clear(B), Snsequence(Pickup(A), Put(A, Table))) Remember (cache) derived propositions. S. C. Shapiro

fa buf @ cse lo Use of Conditional Plan Goal. Plan(Clear(B), Snsequence(Pickup(A), Put(A, Table)))? ? ? SNe. BR to the rescue! S. C. Shapiro

fa buf @ cse lo A FEVAHR Acting Rule all(p, o) ({Agent(p), Thing(o)} &=> {Precondition(Follow(p, o), Near(p, o)), Goal. Plan(Near(p, o), Goto(p, o)), Precondition(Goto(p, o), Lookat(p, o)), Act. Plan(Lookat(p, o), Find(p, o))}) Uses a temporal model. S. C. Shapiro

fa buf @ cse S. C. Shapiro lo Acting According to the Rule

fa buf @ cse lo Acting According to the Rule Follow a red robot. I found a red robot. I am looking at a red robot. S. C. Shapiro

fa buf @ cse lo Acting According to the Rule Follow a red robot. I found a red robot. I am looking at a red robot. I went to a red robot. I am near a red robot. I am following a red robot. S. C. Shapiro

fa buf @ cse lo A Plan for Blowing up UXOs all(a)(Agent(a) => Act. Plan(Blowup(a, UXOs), Act(a, Cascade(Searchfor. Uxo(a), With. Some+(obj, Near(a, obj), With. New({ch ex}, {Charge(ch), Explosion(ex)}, Possess(a, ch), Cascade(Place(a, ch, obj), Hide(a), Waitfor(a, ex), Searchfor. Uxo(a))), goto(a, Safe. Zone)))))) S. C. Shapiro

fa buf @ cse • • lo Outline Introduction Intensional Representation & Propositions SNe. PS Connectives and Quantifiers SNe. RE Acting Constructs Example Plans Representation and Use of Indexicals A Personal Sense of Time Summary S. C. Shapiro

fa buf @ cse lo Representation and Use of Indexicals • Words whose meanings are determined by occasion of use • E. g. I, you, now, then, here, there • Deictic Center <*I, *YOU, *NOW> • *I: SNe. PS term representing Cassie • *YOU: person Cassie is talking with • *NOW: current time. S. C. Shapiro

fa buf @ cse • • S. C. Shapiro lo Analysis of Indexicals (in input) First person pronouns: *YOU Second person pronouns: *I “here”: location of *YOU Present/Past relative to *NOW.

fa buf @ cse lo Generation of Indexicals • *I: First person pronouns • *YOU: Second person pronouns • *NOW: used to determine tense and aspect. S. C. Shapiro

fa buf @ cse lo Use of Indexicals 1 Come here. S. C. Shapiro

fa buf @ cse lo Use of Indexicals 2 Come here. I came to you, Stu. I am near you. S. C. Shapiro

fa buf @ cse lo Use of Indexicals 3 Who am I? Your name is ‘Stu’ and you are a person. Who have you talked to? I am talking to you. Talk to Bill. I am talking to you, Bill. Come here. S. C. Shapiro

fa buf @ cse lo Use of Indexicals 4 Come here. I found you. I am looking at you. S. C. Shapiro

fa buf @ cse lo Use of Indexicals 5 Come here. I found you. I am looking at you. I came to you. I am near you. S. C. Shapiro

fa buf @ cse lo Use of Indexicals 6 Who am I? Your name is ‘Bill’ and you are a person. Who are you? I am the FEVAHR and my name is ‘Cassie’. Who have you talked to? I talked to Stu and I am talking to you. S. C. Shapiro

fa buf @ cse • • lo Outline Introduction Intensional Representation & Propositions SNe. PS Connectives and Quantifiers SNe. RE Acting Constructs Example Plans Representation and Use of Indexicals A Personal Sense of Time Summary S. C. Shapiro

fa buf @ cse lo A Personal Sense of Time • *NOW contains SNe. PS term representing current time. • *NOW moves when Cassie acts or perceives a change of state. S. C. Shapiro

fa buf @ cse lo Representation of Time before after ! after ? ? t n e ag ! time ? ? t even ? ? ? ! before act B 1 n actio obje ct B 6 I lex find S. C. Shapiro NOW

fa buf @ cse lo t 1 NOW S. C. Shapiro Movement of Time before ! after t 2 NOW before ! after t 3 NOW

fa buf @ cse lo t 1 Performing a Punctual Act before ! after t 2 before ! after t 3 time ! event NOW S. C. Shapiro NOW

fa buf @ cse lo t 1 Performing a Durative Act before ! after t 2 time supin t ! subin t t 3 ! event NOW S. C. Shapiro

fa buf @ cse lo The Pacemaker • PML process periodically increments variable COUNT. • *COUNT = some PML integer. • Reset to 0 when NOW moves. • Provides bodily “feel” of passing time. S. C. Shapiro

fa buf @ cse lo Quantizing Time Cannot conceptualize fine distinctions in time intervals. So quantize, e. g. into half orders of magnitude (Hobbs, 2000). S. C. Shapiro

fa buf @ cse lo Movement of Time with Pacemaker ! time t 1 before ! duration after q t 2 KL PML hom NOW COUNT n 0 S. C. Shapiro

fa buf @ cse lo The Problem of the Fleeting Now How can you reason about “now” if it never stands still? S. C. Shapiro

fa buf @ cse lo Fleeting Now Example 1 12: 15: 00: 12: 15: 02: 17: 00: 12: 19: 00: “Is John having lunch now? ” Agent walks to John’s office. Agent sees John at his desk, eating. Agent reports “yes”. Appropriate granularity. S. C. Shapiro

fa buf @ cse lo Fleeting Now Example 2 12: 15: 00: “Is John having lunch now? ” Agent knows John is at home without a phone. Agent contemplates driving to John’s home. Don’t bother---inappropriate granularity. S. C. Shapiro

fa buf @ cse lo The Vagueness of “now” I’m now giving a talk. I’m now teaching a course. I’m now visiting Toronto. I’m now living in Buffalo. The agent is now walking to John’s office. The agent is now seeing if John is eating lunch. Multiple now’s at different granularities. S. C. Shapiro

fa buf @ cse lo NOW-MTF Maximal Temporal Frame based on *NOW Semi-lattice of times, all of which contain *NOW, any of which could be meant by “now” Finite---only conceptualized times of conceptualized states S. C. Shapiro

fa buf @ cse lo Moving NOW with MTF NOW S. C. Shapiro

fa buf @ cse lo Typical Durations “If the walk light is on now, cross the street. ” Relevant duration is typical duration of walk lights. “Is John having lunch now? ” Relevant duration is typical duration of lunch. Use quantized typical durations when updating NOW-MTFs. S. C. Shapiro

fa buf @ cse lo Using Appropriate Granularity Lunch time ? h c n Lu NOW Lu Yes! S. C. Shapiro nc h!

fa buf @ cse • • lo Outline Introduction Intensional Representation & Propositions SNe. PS Connectives and Quantifiers SNe. RE Acting Constructs Example Plans Representation and Use of Indexicals A Personal Sense of Time Summary S. C. Shapiro

fa buf @ cse lo Goal • A computational cognitive agent/robot • That can communicate in natural language. S. C. Shapiro

fa buf @ cse • • lo Intensional Representation & Propositions SNe. PS terms represent mental entities. May assert that two entities are coreferential. Relations/acts may be declared transparent. Propositions are first-class entities. S. C. Shapiro

fa buf @ cse lo SNe. PS Connectives and Quantifiers • Designed logical connectives and rules of inference More appropriate for NLU and Commonsense reasoning than in standard FOPC. S. C. Shapiro

fa buf @ cse lo SNe. RE Acting Constructs • Separate, but Coordinated Syntax and Semantics For Acting and for Reasoning S. C. Shapiro

fa buf @ cse lo Representation and Use of Indexicals • Use of Deictic Center for parser to interpret indexicals as current referents • And for generator to generate indexicals from current referents. S. C. Shapiro

fa buf @ cse lo A Personal Sense of Time • *NOW is current time. • Updated when Cassie acts or perceives a change of state. • Points into MTF to support vagueness of “now”. S. C. Shapiro

fa buf @ cse • • • lo For More Information Personnel Manual Tutorial Bibliography ftp’able SNe. PS source code etc. • http: //www. cse. buffalo. edu/sneps/ S. C. Shapiro