AMM WORKSHOP Control and Representation Vijay Kumar University

AMM WORKSHOP Control and Representation Vijay Kumar University of Pennsylvania John Hollerbach Oussama Khatib Vijay Kumar Al Rizzi Daniela Rus NSF/NASA AMM Workshop March 10 -11, 2005 Houston.

Outline NSF/NASA AMM Workshop State-of-art q Historical perspective (nostalgic memories) Accomplishments in robot control q Summary of last 21 years (WTEC study) q Recent, specific contributions (somewhat biased) Challenges q Panelists Discussion q What are the intellectual problem areas we should address? Infrastructure? Can we can rally around these?

NSF/NASA AMM Workshop Historical Perspective q 40+ years of industrial robotics q >20 years of robotics as an academic discipline q ~13 years of mobile manipulation General Motors Mobility & Manipulation 1961 Unimate 40 years of industrial robotics Rus Sarcos ARC Hollerbach

The Real Agenda for AMM NSF/NASA AMM Workshop Mobility q Unstructured environments Manipulation q Physical interaction with the environment q Closely coupled perception/action q Not physically grounded q Dynamics is important Autonomy q Teleoperation (and therefore haptics) q Supervised Autonomy q Autonomy Haptics John Hollerbach Humanoids Oussama Khatib Perception/Action Al Rizzi Distributed/Modular Daniela Rus

Robotics in the news this week WSJ, 3/7 “…teleoperation with time delays is a vexing problem in robotics…” “…because of the lag, it’s inevitable that the human operator will make tiny errors - errors that will in turn cascade into much bigger ones…” NSF/NASA AMM Workshop

NSF/NASA AMM Workshop Literature Domain q ~8 -10% manipulation q ~3 -4% grasping q ~30 -35% mobility Remaining are on medical, manufacturing, industrial, sensor or “methodology” Control/representation q Model based (~15%) q Data driven approaches (~5%) Counted papers relevant to manipulation and mobility Disclaimer: This is not a scientific study! Conferences surveyed: ICRA 1984 -86, 1998 -2004

Literature (Compared to 1984) Domain q ~10% manipulation (40%) q ~4% grasping q ~35% mobility (4%) NSF/NASA AMM Workshop Total number of papers = 74 Remaining are on medical, manufacturing, industrial, sensor or “methodology” Control/representation q Model based (~15%) (40%) q Data driven approaches (~5%) (3 %) Counted papers relevant to manipulation and mobility Disclaimer: This is not a scientific study! Conferences surveyed: ICRA 1984 -86, 1998 -2004 ~9880 ICRA papers to date

Major Advances Academic/Government Labs NSF/NASA AMM Workshop q Inverse dynamics: application of feedback linearization to serial robots, now routinely used in industrial manipulators (e. g. , ABB) !!! q Time optimal control: along a path subject to dynamics, velocity and acceleration constraints, also used in industrial manipulators !!! q Adaptive robot control: model based adaptive control with global stability guarantee !? q Nonholonomic control: control using time varying feedback or cyclic input, application of differential flat system theory, mostly applied to mobile robots and under-actuated robots. !!! Disclaimer: Not a survey of accomplishments/needs for AMM [Wen and Maciejewski, 04]

Major Advances (Cont. ) NSF/NASA AMM Workshop q Flexible joint robot modeling and control: Application of feedback linearization to flexible joint robots, applied to some industrial arms. ? ! q Teleoperation: wave variable based control for delay robustness. Guarantee stability, but user would feel delayed response. ! q Order N simulation: Application of order N computation to forward and inverse dynamics. Essential for large number degrees of freedom, e. g. , robot with flexible link, micro-robots. !!! q Hybrid force/position, impedance control: Simultaneous regulation of motion and force, applied to machining, assembly, haptic feedback, multi-finger control !!!

AMM Survey (? ) NSF/NASA AMM Workshop ICRA 2000: Grasping and Manipulation Review [Bicchi and Kumar, 2000] Saturation of the area? q. All problems solved q. Not interesting

Two other possibilities NSF/NASA AMM Workshop Problems are too hard Or Nobody is interested in funding this work!

Significant Accomplishments: Industry Technology transfer does happen! Fanuc NSF/NASA AMM Workshop Remember those ~9880 ICRA papers? 20% market share 1800 employees (1300 in research labs, 10 Ph. Ds) 10, 000 robots Technology provides the competitive edge q Before z servo motors/amplifiers q Now z collision detection, compliance control, payload inertia/weight identification, force/vision sensing/integration Ø robots assemble/test robots Ø beyond human performance And mobile manipulation!

Results we can build on… (a parochial view) NSF/NASA AMM Workshop Modeling/controlling humanoids Dynamic manipulation and locomotion Cooperative mobile manipulation Distributed locomotion (and manipulation) systems Haptics and teleoperation

Humanoid dynamics and control Biomechanics for robotics NSF/NASA AMM Workshop Integration (composition) q Realistic models q Integrated control of reach and posture q Minimum principles leading to realistic motions q Task space versus posture space [Khatib]

Humanoid dynamics and control NSF/NASA AMM Workshop Whole-body multi-contact control q Multiple frictional contacts q Models z Posture z Legs z Locomotion [Khatib]

Locomotion and Dexterous Manipulation NSF/NASA AMM Workshop Dynamic manipulation and locomotion q Intermittent interaction q Passive dynamics q Reactive control [Rizzi]

Significant Accomplishments: Academia NSF/NASA AMM Workshop Multiple Mobile Manipulators q Multiple frictional contacts q Maintaining closure [Khatib] [Kumar] [Rus]

M 3 Modular Mobile Manipulation NSF/NASA AMM Workshop Self-organizing, self-assembling, self-repair q Adapt structure q Multiple Functionalities q Can do work [Rus]

Teleoperation and Haptics NSF/NASA AMM Workshop High-DOF telemanipulators Locomotion Interfaces [Hollerbach]

And yet significant challenges remain! NSF/NASA AMM Workshop No successful field deployment of mobile manipulators q Example: Robotic servicing of Hubble (NAS Committee: Brooks, Rock, Kumar) q ETS-VII (JAXA/NASA) z Model-based tele-manipulation z Visual servoing for acquisition of non cooperative targets No robot (product) capable of physical interactions in unstructured environment q Example: Assistive Robotics

Assistive Robotics NSF/NASA AMM Workshop Impact q > 5 million wheelchair users* in the U. S. q > 730, 000 strokes/year (2/3 disabled five years after stroke), > $50 B/year q > 10, 000 SCI/year (most < 20 yrs old) Realistic q Human-in-the-loop q No competing technology z Many other overarching challenges *Inter Agency Working Group on Assistive Technology Mobility Devices

Current technology q Artificial limbs: peg legs, hook hand q Crutches, canes, walkers q Wheelchairs q Environmental control systems q Remote control q Many, many customized products NSF/NASA AMM Workshop

Significant Challenges, Problems NSF/NASA AMM Workshop 1. New hardware, systems 2. Modeling/control 3. Composition, synthesis 4. Model-based versus data-based

p. HRI: Safety and Performance >20 cm compliant covering NSF/NASA AMM Workshop Challenge: 10 x reduction in effective inertia [Khatib]

NSF/NASA AMM Workshop Haptic Interfaces and Mobility Energetic/force interactions between robots and humans q Control simulations or real devices q Personal assist or amplification devices q Rehabilitation or exercise robots Need haptic interfaces that allow manipulation while walking q Psychological argument for VR q Need to control robots that can reach/grasp/manipulate/lean/kick/push [Hollerbach]

Portable Haptic Interfaces Body-worn systems q Powered exoskeleton q Ground-based system with locomotion interface NSF/NASA AMM Workshop

Representation and Control NSF/NASA AMM Workshop q Physics of environmental interaction q Distributed interaction z Whole arm/leg/body q Task representation for non-rigid interaction and manipulation q Control and task allocation of multi-function appendages (feet, legs, hands, arms, etc. ) q Composition of closed-loop (perception/action) behaviors [Rizzi]

Composition of Behaviors: Example NSF/NASA AMM Workshop Four behaviors (closed-loop controllers) q q Pre-shape (open/close) Grasp/release Reach/retract Go to (move)

Composition Pre-shape (close) > Retract NSF/NASA AMM Workshop

Composition Retract > Move NSF/NASA AMM Workshop

Composition Move || Pre-shape (open) NSF/NASA AMM Workshop

Composition Move || Pre-shape (open) NSF/NASA AMM Workshop

Composition Pre-shape (open) > Grasp NSF/NASA AMM Workshop

Composition Grasp > Retract || Move NSF/NASA AMM Workshop

Composition Move NSF/NASA AMM Workshop

Composition Move > Reach > Release NSF/NASA AMM Workshop

Composition NSF/NASA AMM Workshop

Distributed Approaches and Modularity NSF/NASA AMM Workshop Distributed Control q Heterogeneous systems with active modules, passive modules, and tools for mobile manipulation q Mobile sub-assemblies and hierarchical control Thanks to Hod Lipson

Future Concept for Modular Robots in Mobile Manipulation Concept: self-assembly with active grippers and rods Concept: mobile sub-assemblies note: mobile manipulation with dynamic kinematic topology for c-space NSF/NASA AMM Workshop Concept: self-inspection and self-repair with tools

Distributed Approaches and Modularity Challenges NSF/NASA AMM Workshop Control for systems with dynamic kinematic topology q q q Under-constraint systems with continuum of solutions Control for systems with changing c-space Geometrically-driven posture control Control for keeping balance and structural integrity Optimal morphologies for tasks Uncertainty and Error in Modular Systems q Cooperative approach to error recovery in module and structure alignment, connections, assembly, and repair q Dynamical models with uncertainty

Model-based vs. Data Driven NSF/NASA AMM Workshop Control/representation q Model based (~15%) q Data driven approaches (~5%) q Dynamic models are getting more complicated and increasingly sensitive to parameters (uncertainty) q Emphasize completely data-driven approaches

Discussion NSF/NASA AMM Workshop Are there a set of basic research questions that q We can rally around? q Are unique to autonomous mobile manipulation? q Are critical? High-impact? If so, can we create a new research program? q How do we sell it? q How do we take this to the next step? Balance q basic research q high-caliber applied research How do we make robotics a “big science”?

Intellectual Basis for New Program in Autonomous Mobile Manipulation Closed-loop behaviors q Perception-action loops q Vision-based control Composition of behaviors NSF/NASA AMM Workshop Can it be a Tether-esque program? q Sequential q Parallel, hierarchical Task description language q Formal semantics Uncertainty q Understanding and characterizing uncertainty q Data-driven approaches Teleoperation and haptics q Integration mobility with manipulation