Update on RKF progress October 2000 Ken Forbus

Update on RKF progress October, 2000 Ken Forbus Qualitative Reasoning Group Northwestern University

Overview • • Analogical Reasoning Engines Domain Theories Sketching

Our analogical processing tools Base Inputs = propositional descriptions, w/ incremental updates Output = one or two mappings Mappings = correspondences + structural evaluation + candidate inferences SME Target Operates in polynomial time, by exploiting graph labels & greedy algorithms Probe Memory Pool Structure-Mapping Engine provides analogical matching CVmatch SME MAC/FAC provides similarity-based retrieval SME CVmatch Cheap, fast, non-structural SME Output = memory item + SME results No hand-indexing of cases required

How SEQL Works SEQL refines knowledge by progressive alignment of examples New: The GEL algorithm 1. Compare against each generalization Gi. If close enough, assimilate input into Gi by replacing Gi with the overlap of Gi and input and halt. New Stimulus SME Generalizations Exemplars … 2. Compare input against each exemplar Ei. If similar enough, create new generalization from overlap of Ei and input, halt. If nothing similar enough, add input to set of exemplars

Case Mapper: An Analogy GUI • Goal: Provide civilized interface for entering knowledge via analogy – Should be useful platform for experimenting with dialogue moves • Current state – Basic functionality showing signs of life – AI-expert friendly • Next steps – Improved pidgin – Interface to inference machinery for candidate inference evaluation – Explore using dialogue management, simple NLP for interaction

Initial results of Matching

Exploring the candidate inferences

Integrating into the E 2 E system • Strategy: Provide analogy server – KQML for communication – Strategies for analogical reasoning coded in nextgeneration reasoner • Advantages – Neutral with respect to uniprocessor/distributed operation – Enables us to tune our strategies more easily • Drawbacks – Sockets as bottleneck – Need to keep KB in synch • Alternative strategy: Assimilation

Domain Theory Environment (DTE) Reasoner Analogical Reasoner Spatial Reasoner GIS Knowledge Base

Domain Theory Environment (DTE) Uses ODBC, Relational database (Microsoft Access) to store Reasoner KB contents (inspired by Hendler’s PARKA-DB) Analogical Reasoner Spatial Reasoner GIS Knowledge Base

Domain Theory Environment (DTE) Federated architecture, Reasoner supports reasoning sources that provide special-purpose capabilities efficiently Analogical Reasoner Spatial Reasoner GIS Knowledge Base

Domain Theory Environment (DTE) Reasoner Analogical Reasoner Spatial Reasoner GIS Knowledge Base Query-driven backchainer provides basic reasoning services, integration mechanism

Domain Theory Environment (DTE) Reasoner Analogical Reasoner Spatial Reasoner GIS Knowledge Base KQML interface for building servers (e. g. , analogy server, geographic reasoner)

DTE Problems Reasoner High overhead, too many computational Analogical cliffs Reasoner Spatial Reasoner GIS Knowledge Base Too slow, not scaling well

Solution: Build next-generation system • Collaborating with Xerox PARC – John Everett, Reinhard Stolle, Bob Cheslow • Keeping good ideas in DTE: – – Federated architecture/Reasoning sources model Using database to implement KB Query mechanism with simple backchainer as glue Use of LTMS for justifications, reasoning • Overall structure of interfaces to applications using it will be similar • Internals will be very different

Next-generation system Reasoner Knowledge Base Analogical Special-purpose C++ database, Reasoner written by PARC. Built-in support for pattern matching. Adding new. Spatial knowledge: Reasoner DTE DB: 4 assertions/second New DB: 98 assertions/second Retrieval: 2 -3 msec, GISin 111 K assertion KB (preliminary data)

Next-generation System Reasoner Knowledge Base Working memory = LTRE + discrimination tree indexing. Analogical Suggestions Architecture: Reasoner Limit backchaining for “quick” reasoning. Expensive operations. Spatial queued as Reasoner suggestions, processed via agenda mechanism. Multithreaded, to exploit time GIS user spends doing other things. Especially important for sketching, dialogue management

Next-Generation System Streamlined reasoning source interface, with constraint posting for query optimizer. Reasoner Provide qualitative reasoning services by embedding QP theory implementation Create ink-based spatial reasoner, organized for Knowledge incremental processing from Base the ground up Analogical Reasoner Spatial Reasoner Gizmo Mk 2 Perceptual Ink Processor

Current schedule • Halloween: First version turning over • Thanksgiving: DTE applications ported • Christmas: First round of performance tuning finished

Everyday Physical Semantics domain theory • Claim: There is a basic set of physical notions that need to be understood in order to interpret sketched explanations • Surface constraints on motion – Will use Nielsen’s qualitative mechanics • – Collins’ molecular collection ontology – Kim’s bounded stuff ontology – + usual contained stuff ontology – e. g. , Simple notions of surfaces, volumes, forces, and materials • Claim: Qualitative physics research can provide most of this knowledge – Much of it has already been done, in isolated pieces – Needs to be integrated, gaps filled – Tied to sketch-based spatial representations Fluid Ontologies • Surface/fluid interactions – Kim’s qualitative streamline theory • Qualitative topology – Cohn’s spatial algebras • Qualitative Statics – Nielsen & Kim’s qualitative vectors

Multiple Perspectives: An example • How to reason about liquids? • Two models, due to Hayes – Contained stuff ontology: Individuate liquid via the space that it is in. – Piece of stuff ontology: Individuate liquid as a particular collection of molecules.

Fluid ontologies • Contained stuffs – Most detailed: Paper with John Collins, FSThermo domain theory • Pieces of stuff – Molecular collections (w/John Collins) – Plugs (Gordon Skorstad) • Bounded stuffs (H. Kim)

Molecular Collection ontology • Idea: Follow a little piece of stuff around a system – So small that when it reaches a junction, it never splits apart • Provides the perspective gained by tracing through a system of changes

Two containers example

Steam plant example

Refrigerator example

Bounded stuffs • Specialization of contained stuff ontology • Where something is within the space matters – Affects connectivity

Ontology zoo for liquids Contained Stuff Piece of Stuff n co i sit ra Pa Bounded Stuff Molecular Collection Plug

Qualitative Mechanics • Provides axioms for interaction of solids and surfaces not Ok – Qualitative vector representation • Assumes visual parsing of 2 D shapes – Center of gravity, center of rotation critical – Surfaces broken at corners, points of contact Ok not Ok Ok

Qualitative Mechanics • Qualitative angles and vectors • How forces interact with surfaces, constraints on motion • Laminar flow fields

Engineering Thermodynamics • Basics of heat, mass flow • In-depth KB for supporting design, analysis • KB for supporting textbook problem solving – Includes control knowledge, analysis of roles for equations in problem-solving – Pisan’s Ph. D. thesis solves most problems in typical engineering thermodynamics textbooks • Teleological representations for thermodynamic cycles – No chemical interactions

Sketching for knowledge acquisition • s. KEA: Sketch-based Knowledge Entry Associate – Built on top of nu. Sketch + significant extensions – Rich perceptual processing of digital ink • Will support visual analogies and analogies using diagrams Speech I/O and specialized Dialogue Manager • Can be used standalone or as component in larger system RKF Team System • Ink Interpretation is key problem – Collaborating with PARC vision group (Eric Saund, Jim Mahoney) for perceptual processing – Developing domain theories that bridge perception and conceptual knowledge Multimodal Integrator Current Sketches + Interpretations Speech I/O DTE + Evidential Reasoner Graphical Symbology Domain Theory s. KEA Digital ink High-Level Visual Interpreter (Geo. Rep II) Perceptual Ink Processor Everyday Physical Semantics Domain Theory

Tools we will use in sketching Geo. Rep provides high-level visual processing for spatial reasoning Provides bridge between the visual and the conceptual Provides equivalent of Ullman’s universal visual routines Geo. Rep MAGI models processes of symmetry and regularity detection MAGI • Uses variation of structure-mapping laws to detect self -similarity • Same software operates on visual, functional, conceptual, and mathematical representations • Makes predictions consistent with human perceptual data

Visual Symbology domain theory • • Represents conventions for displaying conceptual information graphically State (before) Includes – What visual entities often depict State (after) Process • boxes, blobs, arrows, etc. – Conventional views • side/top/bottom, 2 D/3 D, abstract/physical, cutaways Arg 1 – Conceptual interpretation of visual relations Arg 2 (in-contact (protein-coat virus) (lysosome cell)) • proximity/alignment indicating grouping, inside indicating containment or partonomy, touching indicating contact Cell DNA (Part-of cell. DNA cell) Binary Relationship Virus DNA

Approach: Blob Semantics • Shape, object recognition irrelevant – Linguistic input provides labels and type information – Arrows may be exception wrt recognition • Spatial relationships between blobs is central – Topology • Touching or not, inside, overlap – Proximity • What arrows refer to – Orientation • Multiple reference frames • Quadrant plus relative inclination – Conceptual interpretation of spatial relationships • Hypothesis: Sufficient for – Process diagrams – Action sequences

Issues in blob semantics • Adequacy of visual primitives • User-defined diagram types – Kinds of objects participating – Conceptual interpretation of spatial relationships • Arrow recognition – Support different types of arrows?

Perceptual Ink Processor • Will use next-generation reasoner for conceptual side of reasoning • For visual reasoning, draw on three sources: – Our work on Geo. Rep and Magi (Ferguson’s Ph. D. work) – Eric Saund’s scale-space blackboard (Xerox PARC) • Stroke-based visual routines • Should provide robust proximity detection – Jim Mahoney’s MAPS ideas (Xerox PARC) • Bitmap-based visual routines • Should provide robust qualitative descriptions of free space

Example: e. TDG 10 Map

SR Regions for e. TDG 10 map (hand-sketched)

Hard constraints from SR regions

Voronoi diagram for free space

Junctions provide seeds for open regions

Regions extended from seeds

Edges outside regions form corridor seeds

Combined results for e. TDG 10

Speech or not? • Most multimodal systems use speech recognition – Hands, eyes busy with diagram – Potential problems with speech for RKF • Novel nouns, phrases could lead to distracting speech training during knowledge entry • How open-ended is grammar? Necessity versus user expectations • Trying both in RKF – NLP support with speech • LKB parser (Stanford CSLI) – Experiment: Speechless multimodal interface • Type (or write) label for instance, collection • Draw button, as in nu. Sketch COA Creator • Sacrifice fluidity for expressiveness

Intermediate goal: 1 st generation s. KEA • s. KEA = sketching Knowledge Entry Associate – nu. Sketch application for knowledge formation • Initial targets – Process diagrams – Action sequences • Additional task: Scenario setup for testing everyday physical semantics – Draw examples from biomechanics