Agile Robotics Program Review AR Team MIT Lincoln

Agile Robotics Program Review AR Team MIT, Lincoln Lab, Draper Lab, BAE August 8, 2008

Agile Robotics Team Platform buildup Lenny Paritsky Mitch Leammukda Steve Proulx Rob Truax Ed Corbett Paul Pepin John Leonard Troy Jones Safety & intent USA LIA Whole-SSA simulation Seth Teller Luke Fletcher Situational awareness Lenny Paritsky Missy Cummings Mofe Uku Mitch Leammukda CASCOM Nicholas Roy Seth Teller Mike Boulet Matthew Walter Matt Antone Albert Huang Birsen Donmez Daniela Rus Planning & control DDRE Jake Crandall Matthew Walter Emilio Frazzoli Russ Tedrake Nicholas Roy Jon How Emma Brunskill Jeong hwan Jeon Brandon Luders ARL Supervisor interface Randy Davis Jim Glass Alex Gruenstein Andrew Correa Howard Shrobe Tara Sainath Program support Edmund Williams Nira Manokharan Bryt Bradley Geoff Carrigan Jim Liu Ryne Barry Lee Hetherington Scott Cyphers Stephanie Seneff

Agenda 0900 – 0930: 0930 – 0945: 0945 – 1030: 1030 – 1045: 1045 – 1145: 1200 – 1300: 1300 – 1415: 1415 – 1430: 1430 – 1530: 1530 – 1630: 1630 – 1700: Arrive MIT Kiva room (32 G-449) Informal introductions Summary goals and status Break, walk to demonstration venues Demonstrations (Hangar, Holodeck, Kiva) Lunch (Highlights of other MIT robotics) Technical briefings Break Technical briefings Feedback and discussion Main group adjourns Program management discussion Program review adjourns

Administrative Notes • • • Name placards Room security Kiva conference room Bathrooms Handouts / briefing charts

Agenda 0900 – 0930: Arrive MIT Kiva conference room (32 G-449) 0930 – 0945: Informal introductions 0945 – 1030: Summary goals and status 1030 – 1045: Break, walk to demonstration venues 1045 – 1145: Demonstrations (Hangar, Holodeck, Kiva) 1200 – 1300: Lunch (Highlights of other MIT robotics) 1300 – 1415: Technical briefings 1415 – 1430: Break 1430 – 1530: Technical briefings 1530 – 1630: Feedback and discussion 1630: Main group adjourns 1630 – 1700: Program management discussion 1700: Program review adjourns

Project Goals • Build upon foundation of DARPA Challenge – Real-time perception, planning, and control – Avoid brittle aspects (prior map, GPS) of DGC • Collaborate with LIA, CASCOM – Develop “Agile Robotics” autonomous forklift • Introduce new mobile manipulation aspects – Whole-pallet manipulation (Year 1) – Sub-pallet manipulation (Years 2 -3) • Demonstrate prototype at end of Year 1 • Migrate capability to Army vehicle Years 2, 3

Summary Status • Tackling problem along four fronts, in parallel: – – Requirements analysis: LIA, CASCOM, Ft. Campbell Simulation studies for sensing, dataflow, throughput Mockup experiments, drive-by-wire elements Full-scale prototype development, data collection • At present, substantially ahead of schedule – Several September, December milestones achieved – Demonstrated components in simulation, on mockup – (Partially) working full-scale prototype by fall 2008 • Anticipated fruitful directions for Years 2, 3 – Improved manipulation, reasoning, sensing, scaling

Scenario: A Commandable Forklift (Notional SSA Layout) Reception Pickup Bulk lot 1 1. Autonomous vehicle (flatbed truck) arrives at reception area with two pallets 2. Autonomous forklift meets arriving truck at reception area 3. Forklift unloads one pallet, transports it to specified outdoor bulk lot ASL 4. Forklift unloads second pallet, moves it to queueing area in pickup zone 5. Later, forklift loads queued pallet onto indicated truck in pickup zone 6. Autonomous truck departs pickup zone

Demonstration Scenario Pickup Reception Bulk lot Supervisor Forklift 2 1. Autonomous vehicle (flatbed truck) arrives at reception area with two pallets 2. Autonomous forklift meets arriving truck at reception area 3. Forklift unloads one pallet, transports it to specified outdoor bulk lot ASL 4. Forklift unloads second pallet, moves it to queueing area in pickup zone 5. Later, forklift loads queued pallet onto indicated truck in pickup zone 6. Autonomous truck departs pickup zone

Demonstration Scenario Pickup Reception Bulk lot ASL 3 1. Autonomous vehicle (flatbed truck) arrives at reception area with two pallets 2. Autonomous forklift meets arriving truck at reception area 3. Forklift unloads one pallet, transports it to specified outdoor bulk lot ASL 4. Forklift unloads second pallet, moves it to queueing area in pickup zone 5. Later, forklift loads queued pallet onto indicated truck in pickup zone 6. Autonomous truck departs pickup zone

Demonstration Scenario Pickup Reception Bulk lot QA 4 1. Autonomous vehicle (flatbed truck) arrives at reception area with two pallets 2. Autonomous forklift meets arriving truck at reception area 3. Forklift unloads one pallet, transports it to specified outdoor bulk lot ASL 4. Forklift unloads second pallet, moves it to queueing area in pickup zone 5. Later, forklift loads queued pallet onto indicated truck in pickup zone 6. Autonomous truck departs pickup zone

Demonstration Scenario Pickup Reception Bulk lot 5 1. Autonomous vehicle (flatbed truck) arrives at reception area with two pallets 2. Autonomous forklift meets arriving truck at reception area 3. Forklift unloads one pallet, transports it to specified outdoor bulk lot ASL 4. Forklift unloads second pallet, moves it to queueing area in pickup zone 5. Later, forklift loads queued pallet onto indicated truck in pickup zone 6. Autonomous truck departs pickup zone

Demonstration Scenario Pickup Reception Bulk lot 6 1. Autonomous vehicle (flatbed truck) arrives at reception area with two pallets 2. Autonomous forklift meets arriving truck at reception area 3. Forklift unloads one pallet, transports it to specified outdoor bulk lot ASL 4. Forklift unloads second pallet, moves it to queueing area in pickup zone 5. Later, forklift loads queued pallet onto indicated truck in pickup zone 6. Autonomous truck departs pickup zone

Design Requirements • Develop plausible prototype capability – Compatible with existing Army platforms – Safe, acceptable to Army personnel – Capable of locating, moving, placing pallets – Affordable (acceptable cost increment) – Rugged (operates outdoors on uneven terrain)

Principal Deliverables • Prototype Demonstration (March 2009) – Single voice-, gesture-commandable forklift – Whole-pallet manipulation: truck/ground to ground/truck • Documentation of effort results – Detailed drive-by-wire conversion procedure – Study of task requirements, sensor choices – Report on system architecture and algorithms • Plans for continued capability development – Technology transition plan (Years Two and Three) – Extend capability along several dimensions • Fine-grain manipulation; higher-level reasoning; multiple forklifts and supervisors; adverse environments; GPS-denial

Working Timeline Requirements analysis, system & interface design, safety, user testing Visit to LIA Visit to CASCOM Visit to Ft. Campbell CASCOM Visit to MIT Incursion detection Seamless handoff Shouted warnings Testing & validation Simulation development Port DGC codebase Speech integration Gesture support Pallet detection Situational awareness Whole-SSA simulation Mockup forklift Drive-by-wire prototyping Sensor placement Planning and control Pallet engagement Full-scale prototype Drive-by-wire prototyping Sensor placement (Rented forklift) Planning and control Pallet engagement (Purchased forklift) Pallet mobility Vehicle interaction Capability migration Lincoln Laboratory Kickoff Meeting Jan Feb Mar Apr May Jul Aug Today Year 1 Demo Program Review Jun FMTV studies Sep Oct Nov Dec Jan Feb Year 1 Preliminaries: Drive-by. Wire Platform Development Mar Apr Y 2 Prototype Autonomous Capability Development

Milestone Status ( = completed) • Capability 0. 0: 30 Jun 08 – Analyze and summarize existing SSA operational practices – Demonstrate low-fidelity simulation of terrain, pallets, and trucks – Develop prototype speech, gesture interface to command forklift • Capability 0. 1: 30 Sep 08 – Build a partially-actuated mockup forklift • Gather data on pallet approach, pallet and slot detection / localization – Develop prototype interface, perception, planning, control algorithms – Demonstrate mockup working in a real-world scenario to identify, localize, select, engage, lift, transport, and place one pallet at a time as directed • Capability 0. 2: 31 Dec 08 – Convert a stock manual forklift to drive-by-wire control • Identify model to purchase • Lease example of target forklift • Procure suitable sensor suite and compute server • Instrument leased forklift with sensor suite • Transfer drive-by-wire elements to prototype forklift – Team members complete required OSHA forklift training – Understand degrees of freedom, sensing and control of forklift – Develop functional interface, perception, planning, control algorithms • Capability 0. 3: 31 Mar 09 – Prototype demonstration, venue to be determined (tentatively Ft. Belvoir)

Working Timeline Requirements analysis, system & interface design, safety, user testing Visit to LIA Visit to CASCOM Visit to Ft. Campbell CASCOM Visit to MIT Incursion detection Seamless handoff Shouted warnings Testing & validation Simulation development Port DGC codebase Speech integration Gesture support Pallet detection Situational awareness Whole-SSA simulation Mockup forklift Drive-by-wire prototyping Sensor placement Planning and control Pallet engagement Full-scale prototype Drive-by-wire prototyping Sensor placement (Rented forklift) Planning and control Pallet engagement (Purchased forklift) Pallet mobility Vehicle interaction Capability migration Lincoln Laboratory Kickoff Meeting Jan Feb Mar Apr May Jul Aug Today Year 1 Demo Program Review Jun FMTV studies Sep Oct Nov Dec Jan Feb Year 1 Preliminaries: Drive-by. Wire Platform Development Mar Apr Y 2 Prototype Autonomous Capability Development

Understanding the Task • • MIT team visits LIA, CASCOM, Ft. Campbell Collect, study relevant Dept. of Army pamphlets CASCOM forklift operator visits MIT OSHA forklift operator certification for team members

DGC System Architecture MDF Navigator Sensors RNDF Sensors Goal Local map Drivable Surface, Hazards Situational Planner Perception Drivable surface, Lane markings, Obstacles, Traffic vehicles Trajectory Vehicle Controller Vehicle states Steer, gas, brake Landrover LR 3 Vehicle State Estimator

AR System Architecture • Existing elements (from DGC codebase) – Sensor handlers and terrain perception – Ethernet and CANbus networks • Novel elements – – – – Whole-SSA environment model Interpretation of supervisor speech and gestures Detection of trucks, pallets, loads, pallet slots Forklift mast planning and control Seamless autonomy handoff, return Shouted warning detection Visible and audible apparent intent

AR System Architecture Speech, Gestures Supervisor To bystanders, via annunciators Tablet Supervisor Interface/ Interpreter Success: Confirming Information Mobility and Manipulation Planning and Control Sensing and Situational Awareness Failure: Diagnostic Information Drive-by-Wire Modifications Forklift Operator Standard Forklift

AR System Architecture Speech & Gesture Interpretation Visible, Audible Apparent Intent Navigator Supervisor Interface/ Interpreter Whole-SSA Environment Model Sensors Sensors Sensor data Terrain, Local map People, Trucks, Pallets Perception Vehicle state Vehicle State Estimator Trajectory Vehicle, mast Controller Steer, gas, brake, transmission, parking brake, mast Sensors Sensors Goal Situational Planner Safety: Incursions, Shouted Warnings (Novel elements shown in red boxes) Sensors Microphones Forklift

Working Timeline Requirements analysis, system & interface design, safety, user testing Visit to LIA Visit to CASCOM Visit to Ft. Campbell CASCOM Visit to MIT Incursion detection Seamless handoff Shouted warnings Testing & validation Simulation development Port DGC codebase Speech integration Gesture support Pallet detection Situational awareness Whole-SSA simulation Mockup forklift Drive-by-wire prototyping Sensor placement Planning and control Pallet engagement Full-scale prototype Drive-by-wire prototyping Sensor placement (Rented forklift) Planning and control Pallet engagement (Purchased forklift) Pallet mobility Vehicle interaction Capability migration Lincoln Laboratory Kickoff Meeting Jan Feb Mar Apr May Jul Aug Today Year 1 Demo Program Review Jun FMTV studies Sep Oct Nov Dec Jan Feb Year 1 Preliminaries: Drive-by. Wire Platform Development Mar Apr Y 2 Prototype Autonomous Capability Development

Simulation Studies • Experiments with: – Sensor placement – Truck, pallet, slot detection – SSA structure and layout – Planning, control algorithms – Supervisor interface & bi-directional dataflow

Whole-SSA Simulation • Simulated whole-SSA operation – Multiple forklifts, supervisors, customers • Predict task throughput, wait times for a variety of operating parameters – Speed, # trucks, # supervisors, failure rate &c.

Working Timeline Requirements analysis, system & interface design, safety, user testing Visit to LIA Visit to CASCOM Visit to Ft. Campbell CASCOM Visit to MIT Incursion detection Seamless handoff Shouted warnings Testing & validation Simulation development Port DGC codebase Speech integration Gesture support Pallet detection Situational awareness Whole-SSA simulation Mockup forklift Drive-by-wire prototyping Sensor placement Planning and control Pallet engagement Full-scale prototype Drive-by-wire prototyping Sensor placement (Rented forklift) Planning and control Pallet engagement (Purchased forklift) Pallet mobility Vehicle interaction Capability migration Lincoln Laboratory Kickoff Meeting Jan Feb Mar Apr May Jul Aug Today Year 1 Demo Program Review Jun FMTV studies Sep Oct Nov Dec Jan Feb Year 1 Preliminaries: Drive-by. Wire Platform Development Mar Apr Y 2 Prototype Autonomous Capability Development

Mockup Forklift • Enables experiments with: – Forklift elements to be converted to Drive-by-Wire: • • • Parking brake and latch release Pedal brake Steering wheel and column Mast raise, lower, tilt Mast tine spread and side-shift Rapid transfer to prototype forklift when available – Sensing, planning and control algorithms for: • • • Rear-wheel steering Pallet detection, localization, approach Pallet engagement (tine insertion), transport, placement Unloaded, loaded mobility Incursion detection (operator override)

Working Timeline Requirements analysis, system & interface design, safety, user testing Visit to LIA Visit to CASCOM Visit to Ft. Campbell CASCOM Visit to MIT Incursion detection Seamless handoff Shouted warnings Testing & validation Simulation development Port DGC codebase Speech integration Gesture support Pallet detection Situational awareness Whole-SSA simulation Mockup forklift Drive-by-wire prototyping Sensor placement Planning and control Pallet engagement Full-scale prototype Drive-by-wire prototyping Sensor placement (Rented forklift) Planning and control Pallet engagement (Purchased forklift) Pallet mobility Vehicle interaction Capability migration Lincoln Laboratory Kickoff Meeting Jan Feb Mar Apr May Jul Aug Today Year 1 Demo Program Review Jun FMTV studies Sep Oct Nov Dec Jan Feb Year 1 Preliminaries: Drive-by. Wire Platform Development Mar Apr Y 2 Prototype Autonomous Capability Development

Rental, Purchase Forklift • Enables experiments with: – Sensor placement and sensor data under pallet approach – Electrical interfaces to existing (electrical) DBW elements – Characterization of suspension dynamics when under load – Annunciator visibility/audibility • Purchase forklift as base platform for eventual demonstration prototype – – Toyota 8 FGU-15 lift truck ordered May 19, 2008 Toyota arranged expedited delivery: July 23, 2008 Transfer actuated mechanisms from mockup in August Begin integrated outdoor experiments August 2008

Principal Research Advances • • • Operation in semi-structured environment Hierarchical task-level autonomy Voice/gesture interface from bot’s-eye-view Seamless autonomy handoff & return Annunciation of apparent intent Detection of shouted warnings

Semi-Structured Environment Idea: Give forklift a narrated, guided tour! Manually-driven forklift path Operator utterance and / or gesture

Hierarchical Task-Level Autonomy • Not tele-operated; Not fully autonomous • But: at a useful operating point in between – Summon forklift to working area – Direct it toward one truck (of several) – Direct it toward one aspect (of several) – Direct it toward one pallet (of several) – Help it localize pallet, slots (if necessary) – Direct it to destination (bulk lot, issue area etc. ) • Rich, incremental path to full autonomy

Voice and Gesture Interface • PDA interface for commanding forklift • Interpretation of supervisor’s speech • Supervisor uses stylus gestures to: – Summon and direct forklift – Confirm or edit motion paths – Indicate which pallet, slots are to be engaged

Autonomy Hand-off and Return • Fundamental design constraint: – Forklift pauses if human approaches – Relinquishes autonomy if s/he enters cabin – Operates indistinguishably from manual forklift – Returns to autonomous mode after human exit • Perhaps after an explicit go-ahead from human • Six distinct, independent safety layers – Described later in briefing • Challenging implications for planning layer – Described later in briefing

Apparent Intent • Visible/Audible mode annunciators – Words, symbols, LEDs on cabin exterior – Speakers for audible annunciation – Redundant modes for safety • Robot announces when: – It is about to move – It is stuck and needs help – It is paused, awaiting supervisor’s instructions

Demonstrations 0900 – 0930: Arrive MIT Kiva conference room (32 G-449) 0930 – 0945: Informal introductions 0945 – 1030: Summary goals and status 1030 – 1045: Break, walk to demonstration venues 1045 – 1145: Demonstrations (Hangar, Holodeck, Kiva) 1200 – 1300: Lunch [Highlights of other MIT robotics] 1300 – 1415: Technical briefings 1415 – 1430: Break 1430 – 1530: Technical briefings 1530 – 1630: Feedback and discussion 1630: Main group adjourns 1630 – 1700: Program management discussion 1700: Program review adjourns