Scene Labeling Using Sparse Precision Matrix Nasim Souly

Scene Labeling Using Sparse Precision Matrix Nasim Souly and Dr. Mubarak Shah Center for Research in Computer Vision University of Central Florida CVPR 2016

Introduction Assigning a semantic label to each pixel of an image.

Typical Approach �Segment an image into super-pixels (segments) �Compute local features for each segment and label these using classifiers �Smooth labeling such that neighboring segments receive the same labels �Limitations �unable to incorporate long-range connections �not be able to model the contextual relationships among labels

Our Approach �Model interaction between labels and segments �An energy minimization over a Graph: �Whose structure is captured by Inverse of Covariance matrix (Precision Matrix) �Which encodes only significant interactions �Which avoids a fully connected graph �We use �local and �global information.

Background �

Precision Matrix and Graphical model � Partial correlation is zero if and only if X is conditionally independent of Y given Z, under the Gaussian assumption Covariance Precision Σ =T/k Example from David Mac. Kay's talk on Gaussian Process Basics

Graphical Lasso �

Graphical model and Sparse Precision Matrix �A sample graphical model Blue indicates positive interaction and red negative interaction precision matrix of the ground truth learnt structure from data Images from “Jean Honorio, Luis Ortiz and Dimitris Samaras, Sparse and Locally Constant Gaussian Graphical Models , NIPS 2009”

Proposed Method �Segmentation � Divide image into coherent segments. �Local Classifiers � Compute a features including SIFT, color histogram and etc for each segment. � Use random forest classifiers to classify each segment. �Global Retrieval � Retrieve a subset of the nearest neighbors of the query image from the training data. � Modify Local Classifiers scores leveraging the global GIST features extracted from the data.

Training �Segment training samples �Find features of all segments in dataset �Train local classifiers on super-pixels using Random Forest �Find label graph and pairwise costs between labels for inference Y s are Labels Sparse Inverse of Covariance Data matrix Graphical lasso n images l Labels Build label graph using correlations

Inference � Given a test image find its segments and compute the features � Find the interactions between super-pixels � Obtain unary term using classifier scores and global retrieval � Compute pair-wise cost between selected connections � Optimize the energy function Structure of Graph by Glasso Pair-wise terms by label correlations and Image features segmentation Optimized Solution Unary terms by Classifier (RF) and Retrieval Set

Scene Graph Structure �Capture the structure of the graph for the image segments �Each super-pixel is treated as a random variable �Use graphical lasso and find the partial correlation graph, where the zero indicates no edge �Dependency between super-pixels are obtained

Energy Function Optimization Unary Pairwise Confidence from classifier relevancy of two super-pixels based on their correlations

Experiments and Results Label Graph for SIFTFlow data set Using Empirical Precision matrix Using the sparse partial correlation matrix

Experiments and Results �Stanford-background data set Method Avg Accuracy Local Classifiers 72. 8 Ours (Local Classifiers + Global) 78. 9 Ours (Local + Global + Spatia smoothing 82. 2 Ours Final (sparse structure) 84. 6 Farabet natural [3] 81. 4 Gould [9] 77. 1 Shauai [21] 80. 1

Long distance connections Image Classifier output Spatial Smoothing Our results Meaningful long connection refine the label Ground truth

Experiments and Results �SIFT Flow dataset Method Avg Accuracy Local Classifiers 71. 2 Ours (Local Classifiers + Global) 75. 3 Ours (Local + Global + Spatial smoothing) 77. 7 Ours Final (sparse structure) 80. 6 Farabet [3] 78. 5 Tighe [26] 78. 6 Shauai [21] 80. 1

Relations Between Labels make a difference Image Classifier output Spatial Smoothing Our results Mountain-Road negative correlation Higher cost Building-Door positive correlation Lower-cost Sea-Car Negative correlation Higher cost Ground truth

Summary �We find dependency and interactions between labels as well as super pixels using sparse precision matrix. �Incorporate global information �Taking into account long range relationship �Avoid over smoothing and fully connected graphs �Promising results on different datasets

Thank You!