IIIT Hyderabad Scene Interpretation in images and videos

IIIT Hyderabad Scene Interpretation in images and videos Chetan Jakkoju 200402009 CVIT

Scene interpretation Human can answer: • • • How many taxis ? How many cars ? What type of cars ? How many buildings ? How tall are buildings ? What type of road junction ? IIIT Hyderabad But machine cannot!

Computer vision p r e t e In n o i t a ret Scen IIIT Hyderabad Robotics

What part of image is near or far ? What part of image is at ground ? Some aspects What object is it ? Is it an object? IIIT Hyderabad Where is the object in scene ?

Our interests(1) IIIT Hyderabad • Scene reconstruction ( planar scenes )

Our interests(2) IIIT Hyderabad • Scene recognition ( Outdoor roads )

Piecewise Planar Reconstruction using Convex Optimization IIIT Hyderabad ACCV 2009

Road Map IIIT Hyderabad • • Introduction Applications Existing Solutions & Issues New formulation using Convex Optimization

Introduction Input Output IIIT Hyderabad (Ri, ti) • • Input: Set of images of a piecewise planar scene. Output: 3 D model (normal, perp. distance) and camera parameters (rotation, translation).

IIIT Hyderabad Applications • • Robot navigation Path planning Inserting virtual objects 3 D reconstruction • A. Davison, I. Reid, N. Molton, and O. Stasse. Mono. SLAM: Real-Time Single Camera SLAM. PAMI 2007 • R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, and B. Mac. Intyre. Recent advances in augmented reality. IEEE Computer Graphics and Applications, 21(6): 34– 47, 2001. • N. Snavely, S. M. Seitz, and R. Szeliski. Photo tourism: Exploring photo collections in 3 d. SIGGRAPH 2006.

Homography IIIT Hyderabad • Simple scenario

Existing solutions • SVD based methods (Decompose Homography Matrix) – Faugeras & Zhang methods – Problem: Very much sensitive to noise • Bundle Adjustment methods – Problem: IIIT Hyderabad • Iterative non-linear method • huge time and space requirement apart from correctness.

Our Solution • New formulation in convex optimization framework. • Advantages I. Better solution than Bundle adjustment. II. Standard efficient solvers exist. IIIT Hyderabad (proposed in past 5 years)

Advances in Vision using Convex optimization • Optimization algorithms in Vision (MVG) – Optimal solutions exist for • H from point correspondences • Pose from Essential matrix • Convex optimization is matured enough! • • IIIT Hyderabad • F. Kahl. Multiple view geometry and the l-infinity norm. ICCV 2005 R. Hartley and F. Kahl. Global optimization through searching rotation space and optimal estimation of the essential matrix. ICCV 2007 S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, New York, NY, USA, 2004.

Basic formulation • H matrix • Highly non-linear. • Observation: Fixing pose parameters or plane parameters makes H linear H=[ d. R–tn ] IIIT Hyderabad T

Formulation IIIT Hyderabad • • Given H, decompose it to (n, d) and (R, t). Calculate H != H’ in general Goal: Vary (n, d) and (R, t) so that they close to H

Algorithm • Given H • Decompose H to R, t, n, d • While – Optimize F(n, d) (update n, d) – Optimize F(R, t) (update t) IIIT Hyderabad • end

Extensions • Extension to multiple views • All planes may not be visible in all views! – Sol: We use inter homographies IIIT Hyderabad • ( H 23, H 34, …)

Sample reconstructions Synthetic House showing “visual accuracy” Oxford model house IIIT Hyderabad Baity Hill

IIIT Hyderabad

Summary • Presented convex optimization based algorithm for reconstruction • Applicable for videos. • Synthetic and real experiments show promising results IIIT Hyderabad • Much better optimization frameworks in future.

Part 2 Monocular Terrain recognition IIIT Hyderabad ICPR 2010 & IROS 2010

Problem Classify IIIT Hyderabad • • • Grass Mud Hard mud Road Other

Applications • Autonomous robot navigation • Path planning • Advanced driver assistance systems IIIT Hyderabad Obstacle @ 18 mts Obstacle @ 10 mts

Existing solutions(1) ( In Robotics ) • Solve only sub-problem – Obstacle VS non-obstacle – Use multiple costly sensors • (lasers, ladars etc. , ) IIIT Hyderabad • Though they perform well, they can’t “feel” the terrain surface.

Existing solutions(2) ( In Robotics ) • Good solution is to use IMU sensors – Advantages: • Solve much wider problem of recognizing various types of terrains. – Problems: IIIT Hyderabad • They can only recognize the terrain after they traverse. – “Short-sightedness” • IMU sensors are also costlier.

Ultimate goal • Solving the terrain recognition problem without using costly sensors – Just usingle camera • Advantages: – Light weight – Low power – No “short sightedness” – Direct applications IIIT Hyderabad • in mini-robots • in Driver assistance systems.

Dataset collection IIIT Hyderabad • Camera attached on top of the car

Sample dataset IIIT Hyderabad • 25 videos each of 1 min involving different kind of scenarios

Base method • Prepare Training set and Testing set • In each image, 16 x 16 image block acts as training sample. IIIT Hyderabad • Extract feature-F from the block, and train a classifier-C.

Base method IIIT Hyderabad • Error rates on color features and base classifiers • Naïve Bayes (NB) • Artificial neural networks • K- Nearest neighbours • Support vector machines (linear) (SVM-L) • Support vector machines (Kernel) (SVM-K) • Random forest (RF)

Interesting observations of data • Relative position of different terrains – Eg: Probability of grass area near mud area is greater than that of the grass area near the road area. IIIT Hyderabad • Scale of texture varies majorly in vertical direction.

Proposed method • Previously we trained one classifier on whole image. • Training different classifier on different partition must “capture” the previous observations. IIIT Hyderabad • Note: Partitions increase in squares {22, 32, 42, …}

Experiment-1 ~10 % IIIT Hyderabad • Always decreases the error by ~10%!

Experiment-2 IIIT Hyderabad • Error decreased from 25% to 15%! • (Using 4 -8 classifier sets is desirable)

IIIT Hyderabad Experiment -3 (Smoothness test)

Other enhancement Label Transfer IIIT Hyderabad • Track features from previous frames using optical flow • Transfer the labels • Result: ~45% of image is transferred

Cons IIIT Hyderabad • Memory less • Doesn’t perform well when appearance of terrain varies.

Adaptive algorithm • Track patches in the recent frames. IIIT Hyderabad New training data

IIIT Hyderabad Adaptive algorithm

Experiment • Closed loop test IIIT Hyderabad Road Run 1 • ~5% decrease in error ie, ~20% error rate reduction Run 2

IIIT Hyderabad Demo

Summary • Presented fast-terrain classification method. • Extended the method to adapt online. IIIT Hyderabad • More video processing methods in future.

Conclusions and Future work • New techniques in scene reconstruction and scene recognition. • Reconstruction of piece wise planar scenes. • Main Advantages – All the planes may not be visible in all views. – We also add inter homographies in our framework. • Next we address Terrain recognition. • Own challenging dataset. • We conducted various empirical studies. • Proposed two algorithms IIIT Hyderabad – Partition based method & Adaptive algorithm • Conducted several experiments to validate them.

Conclusions and Future work • Quasi-convex objective functions to Convex objective functions. • Handling outliers • In partition based algorithm, one could replace the simple mode operator with weighted map. IIIT Hyderabad • Adaptive algorithm could be enhanced using -of-the-art semi-supervised ML algorithms. state

Publications • Visesh Chari, Anil Nelakanti, Chetan Jakkoju and C. V. Jawahar. Reconstruction using Convex Optimization. '' ``Piecewise Planar In proceedings of Asian Conference on Computer Vision (ACCV'09). • ``Fast and Spatially-smooth Terrain Classification using Monocular Camera. '' In proceedings of International Conference on Chetan J. , Madhava Krishna and C. V. Jawahar. Pattern Recognition. ( ICPR 2010 ) • ``An Adaptive Outdoor Terrain Classification Methodology using Monocular Camera'' In proceedings of International Chetan J. , Madhava Krishna and C. V. Jawahar. IIIT Hyderabad Conference on Intelligent Robots and Systems. ( IROS 2010 )

Thank you IIIT Hyderabad chetan@research. iiit. ac. in