RobotAssisted Urban Search and Rescue at the WTC

Robot-Assisted Urban Search and Rescue at the WTC Disaster Prof. Robin Murphy Director, Research CRASAR also Associate Prof. , Computer Science & Eng. University of South Florida murphy@csee. usf. edu John Blitch, Jenn Casper, Mark Micire, Brian Minten Feb 19, 2002 www. crasar. org Robot-Assisted Search and Rescue at WTC Disaster (NSF)

NBC Today Show Sept 20, 2001 Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 2

Outline • • Why Robots? What should robots do? What did they do? What is needed? Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 3

The First Known Robot-Assisted USAR • CRASAR stood up by John Blitch Sept. 2 to transfer military robots for USAR: deploy, train, evaluate, & research robots for SAR • At WTC, CRASAR robots and personnel worked with FDNY, NYC, and FEMA teams such as INTF-1, PATF-1, VATF-1, OHTF-1 Duration Tuesday, Sept 11 – Oct. 2 Victims found 10+ CRASAR Teams/people Defense Advanced Projects Research Agency (1), US Dept. of Defense (2), Foster-Miller (5), i. Robot (8), US Navy SPAWAR (3), University of South Florida (4) Robots on site 17, 7 used on rubble pile Field excursions through 9/22… 11 (8 on rubble pile with depths of 20 -45 ft) 1 min set up, avg. 7 minutes per run Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) USF was only USAR certified team USF brought 6, 2 from NSF USF team logged most hours on The Pit in 1 st 2 weeks (except Blitch) 4

photos courtesy of Justin Reuter Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 5

Why Use Robots? • Things that humans can’t do or can’t do safely – “the human use of humans” Norbert Wiener – and that applies to dogs • 135 rescuers died Mexico City, 65 in confined spaces • Not enough trained people – 1 survivor, entombed: 10 rescuers, 4 hours – 1 survivor, trapped/crushed: 10 rescuers, 10 hours Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) photos courtesy of Justin Reuter 6

ESF 9 Tasks: Search, Assessment, Medical Local Jurisdiction FEMA Task Force Organizational Chart FEMA Do. D Liaison Task Force Leader Plan/Ops Safety Search Team Rescue Team Medical Team Logistics Team 2 Canine Search Spec. 2 Rescue Squads 12 Rescue Spec. 2 Medical Specialists Structure Specialist Tech Search Rigging Haz Mat PIO Comm. Spec. Feb 19, 2002 photo. Rescue courtesy of Robot-Assisted Search and at. Justin WTCReuter Disaster (NSF) Logs Spec. 7

Task Highlights Search Team Rescue Team Medical Team 2 Canine Search Spec. Logistics Team Structure Specialist needed not used Tech Search Haz Mat Rescue Phase 9 -11 through 9 -21 worked with FDNY, FEMA found 6+ victims (equivalent to FLTF-2) Feb 19, 2002 Recovery Phase 9 -23 through 10 -02 NYC DDC engineers found 3 -5 victims added Haz. Mat sensors Robot-Assisted Search and Rescue at WTC Disaster (NSF) 8

Search Task Priorities • Rescue first, recover later – first responders (always take care of your own) – civilians in rubble pile – civilians elsewhere • Information for triage – where are the survivors? – where are the people likely to be? – where are the survivors likely to have survived? – which pile of rubble do I remove first? • Typical resources – dogs, search cams, acoustics, sledgehammers Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) dogs: injured by sharp metal, smell only 0. 3 m due to rain on 2 nd day, lack of circulating air Search. Cam: camera on a pole 9

Robots Compared with Search. Cams • Search. Cams ~$10 K, Robots ~$12 K • Search. Cams reach up to 5 meters • Robots reach up to 30 meters, averaged between 6 and 13 meters • Robots can put light on the object, prod it, look at it from different angles • Robots can go through more twisting tunnels • Search. Cams and small robots take <1. 5 minutes to set up and insert camera with zoom, lights couldn’t do this helped to recognize remains of a body Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 10

History of Robot-Assisted USAR in USA Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 11

Timeline of Response (Rescue and Recovery, Inspection not shown) T 11 W 12 Th 13 F S Su M 14 15 16 17 1 S 1 T 18 W 19 4 U Th F S 20 21 22 Inuktuns DARPA USF Foster-Miller Solem rubble pile (The Pit) Foster-Miller Talon Packbot i. Robot Ur. Bot Murphy Casper Micire Minten Blitch Mangolds Haglund Mouru Frost Moore Ciholas Everett Moore Baruch Ciholas/ Laird Alibozek Pratt Schein Smith Levan Hudson Blitch Foster-Miller Feb 19, 2002 Frost Mangolds Pratt Mouru Norman JPO i. Robot Mangolds Mouru Frost Pratt Norman SPAWAR Murphy Casper Micire Minten SPAWAR USF Robot-Assisted Search and Rescue at WTC Disaster (NSF) 12 collateral buildings

Where The Robots Were in the Rubble Pile 9/12/01 FDNY 0 -live 9/16/01 INTF-1 FDNY 0 -live 1 -dead 9/18/01 9/13/01 VATF-1 0 -live 4 -dead FDNY 0 -live 1 -dead Not shown: PATF-1 OHTF-1 NYC 3 2 1 4 5 6 7 New York Times 9/23/01 Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 13

Environment at Ground Zero Personal Safety: -Thick dust for days - asbestos, glass -Rubble largely stable, but if fall, could be impaled - 1 injury: fell out of chair Weather: -45 -70 deg F -rain 2 days, making rubble slick, slippery Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 14

Contextual View: WTC 2 Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 15

Confined Space using Tethered Bots • 2 types (chemical & sewer, HVAC inspection, TRL 9 but not for USAR) – “polymorphic”: ~60 lbs, 7 hours battery, fits in 1 backpack, $12 K • height of mouse to height of squirrel – “fixed geometry”: ~70 lbs. , 7 hours battery, fits in 1 backpack, $10 K • Tethers: 100 -300 ft • Terrain & Environment – voids ~6 -12 inch diameter – dirt, rubble (but not mounds of paper) – inclines: depends • up steep, go with heavier micro. Tracks – water resistant – not intrinsically safe, can melt tracks Inuktun micro. VGTV, Inuktun pipe crawler video, 2 way audio Robot-Assisted Search and Rescue at WTC Disaster (NSF) www. inuktun. com 16 Inuktun micro. Trac video, 2 way audio Feb 19, 2002

Needed Image Processing and Object Recognition Technologies WTC 2, 9/18/01 -tracks of previous robot run -a watch -3, possibly 4 victims (covered in dust, burned) - head, arm socket? , nose, perhaps fingers Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 17

State of Available Information • what viewpoints have already been explored – example: boot? • Feb 19, 2002 no one rewound tape far enough back to catch earlier viewpoint which disambiguated the object! Robot-Assisted Search and Rescue at WTC Disaster (NSF) 18

Confined Space, Structural Assessment using Wireless Bots • 1 type (military hostage rescue, special order, TRL 9 but not for USAR) – backpackable by 2 people, 1 for bot, 1 for OCU & batteries, 12 hours-7 days (standby) – fast, can right itself with practice – can add sensors, payloads – ~$30 K • Wireless – depends on material • Terrain & Environment – – voids: mansize dirt, rubble, inclines: depends water resistant not intrinsically safe FM Solem video, striper, audio www. foster-miller. com Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 19

Need Sensing of State of the Robot, World WTC 4, 9/16/01 • Need state of robot – pose, size relative to the environment – health • some UIs display • fault detection is hard • diagnostics and recovery is slow (replace or 35 minutes) • State of World robot projected a grid to estimate distance – topological vs. metric – 3 D mapping – 3 D interpretation • video overlay • structural reasoning Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 20

Search and Structural Assessment of Collaterally Damaged Buildings • 3 types (military hostage rescue, special order research, TRL 4 -6) ~$30 -45 K – backpackable by 2 people, 1 for bot, 1 for OCU & batteries, 12 hours (Ur. Bot), 20 min (Pack. Bot) – fast, stairs, grasping – self-righting or invertible – can add sensors, payloads, software • Wireless • Terrain & Environment – – buildings, large voids, 3 story drops dirt, rubble, inclines, stairs Packbot is water proof not intrinsically safe SPAWAR Ur. Bot video, audio i. Robot Pack. Bot Feb 19, 2002 FM Talon video, audio, gripper www. spawar. navy. mil, www. irobot. com Robot-Assisted Search and Rescue video, FLIR, 2 way audioat WTC Disaster (NSF) 21

General Mobility & Hardware Problems • Lost 1 Solem robot – lost wireless comms, left in hole, wasn’t there later • Damage – detracked once (high heat) – speared on rubble – just wear and tear • Tethers tangle – only twice not immediately recoverable • 7. 75 “stuck assists” per drop (or once a minute) – but tether handling is significant • 9. 25 “gravity assists” per drop – still have tie a rope around the wireless ‘bots • Wireless and dropouts – can’t compress and do image processing – too numerous/duration to count – make it hard to do image proxy processing • Fear of flipping and getting stuck Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 22

HRI Issues Overall • Warning: camera occlusion 12. 3% of a drop – teleop doesn’t work well in those cases! • Operator errors (Norman ‘ 91) – Mistakes • 2: wrong robot, had to remove and try another – micro. Tracks bulldozes in deep dirt, can’t climb – micro. VGTV too light to get traction • 10% of duration of Inuktun runs spent significant time adjusting lights despite auto gain – need image enhancement – Slips • 0. 25 collisions per drop (oversteer) • 8. 9% of duration, robot wheel slip (high centered, wrong configuration) • Human-Robot Interfaces scared end-users off Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 23

i. Robot: Pack. Bot (experimental) i. Robot Pack. Bot video, FLIR, 2 way audio 2 people, 35 kg • game joystick plus laptop with video & audio • robot state: battery, comms, orientation, camera, encoders • scared off rescuers: too complicated, too long to train Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 24

Timeline of Shift w/ VATF-2 (7: 00 PM 9/18/01 – 7: 00 AM 9/19/01) 1. 2. 3. 4. Drive to Ground 65 min Zero from Javits Center Parts of 9 member ~355 min group splits off while the rest waits 2 operators, 2 robots and VATF-2 search void; Micro. Trac failure 2 operator, 2 robots and VATF-2 search same void; VGTV failure 5. 2 members retrieve spare robot from Javits Center 6. 2 members fix 2 robots using parts from spare robot 7. 9 member group moves to new location to wait 8. Group returns to Javits Center from Ground Zero 12: 00 AM 2 operators return with robots after VATF-2 retreats ~6. 5 min In route ~6. 5 min Waiting 35 min Searching ~5 min ~105 min ~30 min 7: 00 AM 5: 00 AM 3: 00 AM 11: 00 PM 9: 00 PM • 13 minutes for entire shift (green) • No drop >7 minutes • <1. 5 minute set up time or rescuers walk away • 2002 Found 3+ victims Feb 19, Robot-Assisted Search and Rescue at WTC Disaster (NSF) 25

Other problems: Asynergism • Lack of interoperability / “on the other robot” – Image processing, intelligent assistance – Software, sensors couldn’t be migrated between platforms (“plug and perceive”) – Only the “dumbest” robots were well-used • Known capabilities “lost” or “back in the lab” • • robots as wireless repeaters self righting self reacquistion of comms signal general office navigation, obstacle avoidance Demo, hardware focus, rather than Systems focus Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 26

Problems as Point of Departure for Theory • USAR robots can be generic, adapted in the field for the particular task, thought of a system • Sensors and sensing need to be improved, miniaturized, $$ – State of robot: highly sensate robots, health monitoring – State of world: 3 D maps plus understanding structure – State of available information • Mobility and hardware need to be improved – Polymorphic and elephant trunk sensors masts – Teams of robots • marsupialism for delivery, relay, proxy processing • physically coupled: tether managers, in-line collaborative teleoperation • distributed: hybrid tethers • Human-robot Interaction needs to be improved – Perceptual user interfaces not well-defined, mapping & human models of situational awareness – Training and access is an issue; over the internet Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 27

Robot-Assisted USAR as IT CONNECTIVITY HOSPITAL MOBILITY DECIDABILITY UNIVERSITY PERCEPTIBILITY HOT ZONE Search Teams WARM ZONE COLD ZONE Base of Operations Command Post Team Task Leader& Staging Area personnel prep Incident Commander, Forward Station Structural Engineers, Section Chief Robot Specialists USABILITY CRASAR, GOVT LABS NEW-TECH LABS ADAPTABILITY Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 28

Summary of Performance • Robots were successfully used at the World Trade Center – Success: quick response (luck) – effective performance • 12/11 equal to manual technical search for 24/10 days • user acceptance – Training in Dec. 02 for East Coast firefighters – Training in Apr. 02 at FDIC in Indianapolis – ~30 copies of data sets to fire rescue teams to date • USF as a university provided expertise from field research, managed data collection (and pushing data dissemination) Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 29

However, We Could’ve Done More… • Had more robots, sensors, and people than were utilized due to credentialing, organizational issues – Lost days 3, 4 – Never saw a 24/7 deployment cycle – Only 75% of the available Inuktuns were deployed at any given time, despite effectiveness – Buildings could have been surveyed, if necessary, stored on videotape • Capabilities never exploited – Robots could place tubing to transport air/water/meds to survivors • Dr. Eric Rasmussen, 3 rd Fleet Surgeon, sent medical equipment – Added air quality monitors on larger robots after 9/25 – AI Software on larger robots not interoperable with smaller and newer robots Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 30

Robots Can’t Replace the Real Heroes more information and public video & stills: www. crasar. org murphy@csee. usf. edu Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 31