Some Reflections on Augmented Cognition Eric Horvitz ISAT

Some Reflections on Augmented Cognition Eric Horvitz ISAT & Microsoft Research November 2000

Toward Augmented Cognition Challenge: Volume of information, overall complexity of defense tasks continue to grow at a rapid pace, in stark contrast to human cognitive abilities, which remain static: Attention Ø Memory Ø Learning Ø Comprehension Ø Sensory bandwidth Ø Visualization abilities Ø Judgment and decision making Ø In contrast, computational capabilities have continued to grow rapidly Apply computation to support / augment human cognition

Augmented Cognition Developing new powers of understanding, remembering, and decision making via new technologies custom-tailored for human— computer collaboration, symbiosis Methods, designs that harness computation and explicit knowledge about human limitations to open bottlenecks, address biases, deficits in cognition. u Techniques and interfaces that allow for fluid mixedinitiative interaction to solve problems. u Continual background sensing, learning, inference to understand trends, patterns, situations relevant to a user’s context and goals. u

Modern Psychology: Cognition as information processing u u u Human cognition: Vast abilities coupled with highly characterized / characterizable limitations & bottlenecks Mind as an architecture composed of different subsystems that form, remember, transmit representations of the world. Work on distinct dimensions of cognition: Attention, memory, learning, concept attainment, visualization, judgment

Timing: Is Now the Time to Undertake Such an Effort? Understanding of human as resource-limited reasoner, well-characterized bottlenecks, capabilities, biases • Limitations in serial processing • Attention • Concept attainment • Learning and memory • Visualization • Judgment and decisions • Collaboration, group decisions • Psychophysics • HCI expertise, results Results in Cognitive Psychology

Timing: Is Now the Time to Undertake Such an Effort? Leaps in computation, memory, algorithmic prowess • Raw computational resources (CPU, memory) • Connectivity • Sensing • Sensor fusion • Inference and classification • Learning • User modeling • Speech recognition • Vision • Interaction, input • UI, display design Computational power

Augmented Cognition Efforts u u Characterizing inefficiencies, problems, opportunities re: performing critical tasks based on fundamental cognitive limitations Identify, apply results from cognitive psychology to build computation solutions Identify new cognitive psychology research Identify new joint CS—cognitive psychology efforts Cognitive Tasks Existing Psychological Results on Cognitive Limitations Memory Concept Computation Divided Attainment Attention Abilities & efficiencies Target

Augmented Cognition Efforts u u Characterizing inefficiencies, problems, opportunities re: performing critical tasks based on fundamental cognitive limitations Identify, apply results from cognitive psychology to build computation solutions Identify new cognitive psychology research Identify new joint CS—cognitive psychology efforts Cognitive Tasks * New. Existing Cog. Psych. Research Psychological Results on * HCI, Aug. Cognition Cognitive Limitations Research ? Memory Concept Divided Attainment Attention Abilities & efficiencies Target

Augmented Cognition: Opportunities Ø Enhance learning and memory via reminder systems & methods Ø Automate specific aspects of problem solving & filtering Ø Modulate / triage communications Ø Develop new visualizations, other info. rendering to increase rate of “concept attainment”---raising effective human-computer bandwidth

Augmented Cognition: Opportunities Ø Support sensory fusion, judgment, action under uncertainty Ø Provide vigilant monitoring for situations requiring attention Ø Monitoring, knowledge about human and machine errors Ø Extend abilities to coordinate, collaborate with other people and systems Ø Design multimodal, context-aware systems for more efficient, natural human—computer collaboration

Low-Hanging(? ) Fruit u u u u u Information filtering and triage Mixed-initiative interaction Context-sensitive UI / computing Intelligent reminding Managing attention and disruption Visualization & comprehension Human-error--aware systems Automated sensor fusion Judgment de-biasing systems

Risks in Building a Strong, Valuable Program u Timing: Too early…? Ø Ø u Strength of research Ø Ø u Need more complete understanding of human bottlenecks in real world? Need more sophisticated computational procedures? Rigorous, foundational work may be needed Poor initial stabs may lead to premature discontinuation, disappointment Focus Ø Ø Choosing some foci for early traction Avoiding replication of work in commercial sector

Several Sample Projects u Vista: Models for limiting, controlling u Handsfree decision support: Fluid u Lumiere: Context- and competence-based u Lookout: Mixed-initiative interaction. u Priorities: Automated assignment of urgency, information based on coarse value of revealed information. interaction for providing recommendation, updates; context-sensitive speech. assistance alerting, communication, caching by value.

Several Sample Projects u Notification Platform: Attention in HCI, u Continual computation: Use background u Learning information goals: Learning about a u Qualia: Models of attention, perception in u Context-aware devices: Enhance UI with notification architectures, UI, psych studies of disruption, reminders. computation to forecast goals, attention; guide precomputing, caching. Background query, document tracking. user’s goals from behavior, log files. multimedia rendering and communication implicit gestures, context-sensing.

Project Drill Down & Demos