RealTime Vision on a Mobile Robot Platform Mohan

Real-Time Vision on a Mobile Robot Platform Mohan Sridharan Joint work with Peter Stone The University of Texas at Austin smohan@ece. utexas. edu

Motivation ¡ Computer vision challenging. l l ¡ “State-of-the-art” approaches not applicable to real systems. Computational and/or memory constraints. Focus: efficient algorithms that work in real -time on mobile robots.

Overview ¡ Complete vision system developed on a mobile robot. ¡ Challenges to address: l l ¡ Color Segmentation. Object recognition. Line detection. Illumination invariance. On-board processing– computational and memory constraints.

Test Platform – Sony ERS 7 ¡ ¡ ¡ 20 degrees of freedom. Primary sensor – CMOS camera. IR, touch sensors, accelerometers. Wireless LAN. Soccer on 4. 5 x 3 m field – play humans by 2050!

The Aibo Vision System – I/O ¡ Input: Image pixels in YCb. Cr Color space. l l Frame rate: 30 fps. Resolution: 208 x 160. ¡ Output: Distances and angles to objects. ¡ Constraints: l l On-board processing: 576 MHz. Rapidly varying camera positions.

Robot’s view of the world…

Vision System – Flowchart…

Vision System – Phase 1: Segmentation. ¡ Color Segmentation: l l Hand-label discrete colors. Intermediate color maps. NNr weighted average – Master color cube. 128 x 128 color map – 2 MB.

Vision System – Phase 1: Segmentation. ¡ ¡ Use perceptually motivated color space – LAB. Offline training in LAB – generate equivalent YCb. Cr cube.

Vision System – Phase 1: Segmentation.

Vision System – Phase 1: Segmentation. ¡ ¡ ¡ Use perceptually motivated color space – LAB. Offline training in LAB – generate equivalent YCb. Cr cube. Reduce problem to table lookup. l Robust performance with shadows, highlights. l YCb. Cr – 82%, LAB – 91%.

Sample Images – Color Segmentation.

Sample Video – Color Segmentation.

Some Problems… ¡ Sensitive to illumination. l l ¡ Frequent re-training. Robot needs to detect and adapt to change. Off-board color labeling – time consuming. l Autonomous color learning possible…

Vision System – Phase 2: Blobs. ¡ Run-Length encoding. l ¡ Region Merging. l l ¡ Starting point, length in pixels. Combine run-lengths of same color. Maintain properties: pixels, runs. Bounding boxes. l l Abstract representation – four corners. Maintains properties for further analysis.

Sample Images – Blob Detection.

Vision System – Phase 2: Objects. ¡ Object Recognition. l l ¡ Heuristics on size, shape and color. Previously stored bounding box properties. Domain knowledge. Remove spurious blobs. Distances and angles: known geometry.

Sample Images – Objects.

Vision System – Phase 3: Lines. ¡ ¡ ¡ Popular approaches: Hough transform, Convolution kernels – computationally expensive. Domain knowledge. Scan lines – greenwhite transitions – candidate edge pixels.

Vision System – Phase 3: Lines. ¡ Incremental least square fit for lines. l l ¡ ¡ Efficient and easy to implement. Reasonably robust to noise. Lines provide orientation information. Line Intersections can be used as markers. l l Inputs to localization. Ambiguity removed through prior position knowledge.

Sample Images – Objects + Lines.

Some Problems… ¡ Systems needs to be re-calibrated: l l ¡ Re-calibration very time consuming. l ¡ Illumination changes. Natural light variations: day/night. More than an hour spent each time… Cannot achieve overall goal – play humans. l That is not happening anytime soon, but still…

Illumination Sensitivity – Samples. ¡ Trained under one illumination: ¡ Under different illumination:

Illumination Sensitivity – Movie…

Illumination Invariance - Approach. ¡ Three discrete illuminations – bright, intermediate, dark. ¡ Training: l l l Performed offline. Color map for each illumination. Normalized RGB (rgb – use only rg) sample distributions for each illumination.

Illumination Invariance – Training. ¡ Illumination: bright – color map

Illumination Invariance – Training. ¡ Illumination: bright – map and distributions.

Illumination Invariance – Training.

Illumination Invariance – Testing.

Illumination Invariance – Testing.

Illumination Invariance – Testing.

Illumination Invariance – Testing.

Illumination Invariance – Testing. ¡ Testing - KLDivergence as a distance measure: l l l ¡ Robust to artifacts. Performed on-board the robot, about once a second. Parameter estimation described in the paper. Works for conditions not trained for… l Paper has numerical results.

Adapting to Illumination changes – Video

Some Related Work… ¡ CMU vision system: Basic implementation. l ¡ German Team vision system: Scan Lines. l ¡ James Bruce et al. , IROS 2000 Rofer et al. , Robo. Cup 2003 Mean-shift: Color Segmentation. l Comaniciu and Peer: PAMI 2002

Conclusions ¡ A complete real-time vision system – on board processing. ¡ Implemented new/modified version of vision algorithms. ¡ Good performance on challenging problems: segmentation, object recognition and illumination invariance.

Future Work… ¡ Autonomous color learning. l ¡ ¡ ¡ AAAI-05 paper available online. Working in more general environments, outside the lab. Automatic detection of and adaptation to illumination changes. Still a long way to go to play humans .

Autonomous Color Learning – Video ¡ More videos online l www. cs. utexas. edu/~Austin. Villa/

THAT’S ALL FOLKS www. cs. utexas. edu/~Austin. Villa/

Question – 1: So, what is new? ? ¡ ¡ ¡ Robust color space for segmentation. Domain-specific object recognition + line detection. Towards illumination invariance. Complete vision system – closed loop. Accept – cannot compare with other teams, but overall performance good at competitions…

Vision – 1: Why LAB? ? ¡ ¡ Robust color space for segmentation. Perceptually motivated. Tackles minor changes – shadows, highlights. Used in robot rescue…

Vision – 2: Edge pixels + Least Squares? ? ¡ ¡ Conventional approaches time consuming. Scan lines faster: l l ¡ Reduces colors needing bounding boxes. LS easier to implement – fast too. Accept – have not compared with any other method…

Vision – 3: Normalized RGB ? ? ¡ ¡ YCb. Cr separates luminance – but not good for practice on Aibo. Normalized RGB (rgb): l l ¡ Reduces number of dimensions - storage. More robust to minor variations. Accept – have compared with YCb. Cr alone – LAB works but more storage and calculations…

Illumination Invariance – Training.

Illumination Invariance – Testing.