OfftheShelf VisionBased Mobile Robot Sensing Zhichao Chen Advisor

Off-the-Shelf Vision-Based Mobile Robot Sensing Zhichao Chen Advisor: Dr. Stan Birchfield Clemson University

Vision in Robotics • A robot has to perceive its surroundings in order to interact with it. • Vision is promising for several reasons: Ø Non-contact (passive) measurement Ø Low cost Ø Low power Ø Rich capturing ability

Project Objectives Path following: Traverse a desired trajectory in both indoor and outdoor environments. 1. “Qualitative vision-based mobile robot navigation”, Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2006. 2. “Qualitative vision-based path following”, IEEE Transactions on Robotics, 25(3): 749 -754, June 2009. Person following: Follow a person in a cluttered indoor environment. “Person Following with a Mobile Robot Using Binocular Feature-Based tracking”, Proceedings of the IEEE International Conference on Intelligent Robots and Systems (IROS), 2007 Door detection: Build a semantic map of the locations of doors as the robot drives down a corridor. “Visual detection of lintel-occluded doors from a single camera”, IEEE Computer Society Workshop on Visual Localization for Mobile Platforms (in association with CVPR), 2008,

Motivation for Path Following • Goal: Enable mobile robot to follow a desired trajectory in both indoor and outdoor environments • Applications: courier, delivery, tour guide, scout robots • Previous approaches: • • Image Jacobian [Burschka and Hager 2001] Homography [Sagues and Guerrero 2005] Homography (flat ground plane) [Liang and Pears 2002] Man-made environment [Guerrero and Sagues 2001] Calibrated camera [Atiya and Hager 1993] Stereo cameras [Shimizu and Sato 2000] Omni-directional cameras [Adorni et al. 2003]

Our Approach to Path Following • Key intuition: Vastly overdetermined system (Dozens of feature points, one control decision) • Key result: Simple control algorithm – – Teach / replay approach using sparse feature points Single, off-the-shelf camera No calibration for camera or lens Easy to implement (no homographies or Jacobians)

current image milestone image overview top-down view Preview of Results

Tracking Feature Points Kanade-Lucas-Tomasi (KLT) feature tracker • Automatically selects features using eigenvalues of 2 x 2 gradient covariance matrix gradient of image • Automatically tracks features by minimizing sum of squared differences (SSD) between consecutive image frames unknown displacement gray-level images • Augmented with gain and bias to handle lighting changes • Open-source implementation [http: //www. ces. clemson. edu/~stb/klt]

Teach-Replay track detect features goal feature destination initial feature Teaching Phase start compare track features Replay Phase current feature goal feature

Qualitative Decision Rule Landmark image plane feature Robot at goal u. Goal u. Current funnel lane Feature is to the right |u. Current| > |u. Goal| “Turn right” No evidence “Go straight” Feature has changed sides sign(u. Current) ≠ sign(u. Goal) “Turn left”

The Funnel Lane at an Angle Landmark α image plane feature Robot at goal α α funnel lane Feature is to the right “Turn right” No evidence “Go straight” Side change “Turn left”

A Simplified Example Landmark feature Robot at goal funnel lane “Go straight” “Turn right” “Turn left”

The funnel Lane Created by Multiple Feature Points Landmark #2 Landmark #1 Landmark #3 α α Feature is to the right “Turn right” No evidence “Do not turn” Side change “Turn left”

Qualitative Control Algorithm Funnel constraints: u. Goal Desired heading where φ is the signed distance between the u. C and u. D

Incorporating Odometry Desired heading from ith feature from odometry point N is the number of the features;

Overcoming Practical Difficulties To deal with rough terrain: Prior to comparison, feature coordinates are warped to compensate for a non-zero roll angle about the optical axis by applying the RANSAC algorithm. To avoid obstacles: The robot detects and avoids an obstacle by sonar, and the odometry enables the robot to roughly return to the path. Then the robot converges to the path using both odometry and vision.

current image milestone image overview top-down view Experimental Results Videos available at http: //www. ces. clemson. edu/~stb/research/mobile_robot

Experimental Results: Rough Terrain

Experimental Results: Avoiding an Obstacle

Experimental Results Indoor Outdoor Imaging Source Firewire camera Logitech Pro 4000 webcam

Project Objectives Path following: Enable mobile robot to follow a desired trajectory in both indoor and outdoor environments. 1. “Qualitative vision-based mobile robot navigation”, Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2006. 2. “Qualitative vision-based path following”, IEEE Transactions on Robotics, 2009 Person following: Enable a mobile robot to follow a person in a cluttered indoor environment by vision. “Person Following with a Mobile Robot Using Binocular Feature-Based tracking”, Proceedings of the IEEE International Conference on Intelligent Robots and Systems (IROS), 2007 Door detection: Detect doors as the robot drives down a corridor. “Visual detection of lintel-occluded doors from a single camera”, IEEE Computer Society Workshop on Visual Localization for Mobile Platforms (in association with CVPR), 2008

Motivation • Goal: Enable a mobile robot to follow a person in a cluttered indoor environment by vision. • Previous approaches: • Appearance properties: color, edges. [Sidenbladh et al. 1999, Tarokh and Ferrari 2003, Kwon et al. 2005] § Person has different color from background or faces camera. § Lighting changes. • Optical flow. [Piaggio et al 1998, Chivilò et al. 2004] § Drift as the person moves with out-of-plane rotation • Dense stereo and odometry. [Beymer and Konolige 2001] § difficult to predict the movement of the robot (uneven surfaces, slippage in the wheels).

Our approach • Algorithm: Sparse stereo based on Lucas-Kanade feature tracking. • Handles: • Dynamic backgrounds. • Out-of-plane rotation. • Similar disparity between the person and background. • Similar color between the person and background.

System overview

Detect 3 D features of the scene ( Cont. ) • Features are selected in the left image IL and matched in the right image IR. The size of each square indicates the horizontal disparity of the feature. Left image Right image

System overview

Detecting Faces • The Viola-Jones frontal face detector is applied. • This detector is used both to initialize the system and to enhance robustness when the person is facing the camera. Note: The face detector is not necessary in our system.

Overview of Removing Background 1) using the known disparity of the person in the previous image frame. 2) using the estimated motion of the background. 3) using the estimated motion of the person

Remove Background Step 1: Using the known disparity Discard features for which where and is the known disparity of the person in the previous frame, is the disparity of a feature at time t. Original features Foreground features in step 1 Background features

Remove Background Step 2: Using background motion • Estimate the motion of the background by computing a 4 × 4 affine transformation matrix H between two image frames at times t and t + 1: • Random sample consensus (RANSAC) algorithm is used to yield dominant motion. Foreground features with similar disparity in step 1 Foreground features after step 2

Remove Background Step 3: Using person motion • Similar to step 2, the motion model of the person is calculated. • The size of the person group should be the biggest. • The centroid of the person group should be proximate to the previous location of the person. Foreground features after step 2 Foreground features after step 3

System overview

Experimental Results

Video

Project Objectives Path following: Enable mobile robot to follow a desired trajectory in both indoor and outdoor environments. 1. “Qualitative vision-based mobile robot navigation”, Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2006. 2. “Qualitative vision-based path following”, IEEE Transactions on Robotics, 2009 Person following: Enable a mobile robot to follow a person in a cluttered indoor environment by vision. “Person Following with a Mobile Robot Using Binocular Feature-Based tracking”, Proceedings of the IEEE International Conference on Intelligent Robots and Systems (IROS), 2007 Door detection: Detect doors as the robot drives down a corridor. “Visual detection of lintel-occluded doors from a single camera”, IEEE Computer Society Workshop on Visual Localization for Mobile Platforms (in association with CVPR), 2008

Motivation for Door Detection Metric map Topological map Either way, doors are semantically meaningful landmarks

Previous Approaches to Detecting Doors Range-based approaches sonar [Stoeter et al. 1995], stereo [Kim et al. 1994], laser [Anguelov et al. 2004] Vision-based approaches fuzzy logic neural network color segmentation [Munoz-Salinas et al. 2004] [Cicirelli et al 2003] [Rous et al. 2005] Limitations: • require different colors for doors and walls • simplified environment (untextured floor, no reflections) • limited viewing angle • high computational load • assume lintel (top part) visible

What is Lintel-Occluded? Lintel-occluded § post-and-lintel architecture § camera is low to ground § cannot point upward b/c obstacles lintel post

Our Approach Assumptions: • Both door posts are visible • Posts appear nearly vertical • The door is at least a certain width Key idea: Multiple cues are necessary for robustness (pose, lighting, …)

Video

Pairs of Vertical Lines vertical lines Canny edges detected lines non-vertical lines 1. 2. 3. 4. 5. Edges detected by Canny Line segments detected by modified Douglas-Peucker algorithm Clean up (merge lines across small gaps, discard short lines) Separate vertical and non-vertical lines Door candidates given by all the vertical line pairs whose spacing is within a given range

Homography In the image In the world (x’, y’) (x, y)

Prior Model Features: Width and Height Principal point ? ? (0, y) (0, 0) (x, y) (x, 0)

An Example As the door turns, the bottom corner traces an ellipse (projective transformation of circle is ellipse) But not horizontal

Data Model (Posterior) Features Image gradient along edges (g 1) texture (g 5) Placement of top and bottom edges (g 2 , g 3) Kick plate (g 6) Color (g 4) Vanishing point (g 7) and two more…

Data Model Features (cont. ) Intensity along the line darker (light off) positive brighter (light on) negative no gap Bottom gap(g 8)

Data Model Features (cont. ) Slim “U” vertical door lines door wall Lleft intersection line of wall and floor wall bottom door edge extension of intersection line Concavity(g 9) LRight ε floor

Two Methods to Detect Doors Training images Adaboost Weights of features The strong classifier Weights of features Bayesian formulation (yields better results)

Bayesian Formulation door image Taking the log likelihood, Data model Prior model

MCMC and DDMCMC • Markov Chain Monte Carlo (MCMC) is used here to maximize probability to detect door (like random walk through state space of doors) • Data driven MCMC (DDMCMC) is used to speed up computation § doors appear more frequently at the position close to the vertical lines § the top of the door is often occluded or a horizontal line closest to the top § the bottom of the door is often close to the wall/floor boundary.

Experimental Results: Similar or Different Door/Wall Color

Experimental Results: High Reflection / Textured Floors

Experimental Results: Different Viewpoints

Experimental Results: Cluttered Environments

Results 25 different buildings 600 images: • 100 training • 500 testing 91. 1% accuracy with 0. 09 FP per image Speed: 5 fps 1. 6 GHz (unoptimized)

False Negatives and Positives strong reflection concavity and bottom gap tests fail distracting reflection two vertical lines unavailable concavity erroneously detected distracting reflection

Navigation in a Corridor • Doors were detected and tracked from frame to frame. • Fasle positives are discarded if doors were not repeatedly detected.

Video

Conclusion • Path following • • • Teach-replay, comparing image coordinates of feature points (no calibration) Qualitative decision rule (no Jacobians, homographies) Person following • • Detects and matches feature points between a stereo pair of images and between successive images. RANSAC-based procedure to estimate the motion of each region Does not require the person to wear a different color from the background. Door detection • • Integrate a variety of features of door Adaboost training and DDMCMC.

Future Work • Path following § § • Incorporating higher-level scene knowledge to enable obstacle avoidance and terrain characterization Connecting multiple teaching paths in a graph-based framework to enable autonomous navigation between arbitrary points. Person following § Fusing the information with additional appearance-based information ( template or edges). § Integration with EM tracking algorithm. • Door detection § § Calibrate the camera to enable pose and distance measurements to facilitate the building of a geometric map. Integrated into a complete navigation system that is able to drive down a corridor and turn into a specified room