Agile Robotics PDA Interface Randall Davis Jim Glass

Agile Robotics: PDA Interface Randall Davis, Jim Glass, Andrew Correa, Scott Cyphers, Alex Gruenstein, Lee Hetherington, Tara Sainath MIT CSAIL

Natural Interaction n Eventually: natural speech, sketching, hand gestures n Near-term q basic speech grammar q small graphical vocabulary of sketch gestures 12/20/2021 2

Perception at the User Interface n Hear and understand q q n Understand written gestures q q n PDA-based speech commands for routine operation Robot-based hearing for safety Indicate pallet; path to follow; areas to avoid; Communicate even in a noisy environment Challenges q q Dynamic noise in the environment (forklift , multiple speakers, other vehicles, etc. ) Interaction of pen with the world 12/20/2021 3

Basic Interface Capabilities n Speech Basic command control of robot functions, e. g. q § § § “Bot, come to reception” “Lift the red pallet and move it to the pickup area” “Approach this pallet and move it to storage Bravo Charlie” q Currently implemented as context-free grammar q Conveys understood concepts while listening q Interacts with sketching interpretation 12/20/2021 4

Speech Grammar n Grammar q <sentence> = [Bot] ( <stop> | <summon> | <approach> | <lift> ) q <stop> q <summon> = Come <location> q <approach> = Approach ( that | this | the | [<color>] ) pallet q <color> q <lift> q <location> = here | to [the] ( reception | queuing | pickup | issue) [area] | to storage <letter> q <letter> = alpha | bravo | charlie | delta = Stop | Cancel = red | blue | green | purple | yellow | orange = [and] ( lift | move ) ( that pallet | it) <location> 12/20/2021 5

Basic Interface Capabilities n Sketching q Select Pickup Reception Bulk lot q Avoid q Path q Destination 12/20/2021 6

Current Architecture speech server planner / simulator speech interp. web server ink interp. inking, recording 12/20/2021 7

Architecture: Sketching speech server planner / simulator speech interp. web server ink interp. 12/20/2021 8

Architecture: Speech server planner / simulator speech interp. web server ink interp. “come to storage area charlie bravo” command=summon destination=storage letter 1=C letter 2=B 12/20/2021 9

Architecture speech server planner / simulator speech interp. web server ink interp. inking, recording 12/20/2021 10

Demo

Timeline apr 08 aug 08 nov 08 feb 09 apr 09 basic speech grammar simple sketch recog. speech and sketch interp onboard sketch grounded in world model enhanced speech detection and understanding sketch interp in context of world model sound interpretation for situation awareness sketch influences future perception 12/20/2021 14