A CRITICAL VIEW OF CONTEXT Biologically Inspired Models

A CRITICAL VIEW OF CONTEXT Biologically Inspired Models of Vision course Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos

INTRODUCTION • Extraction Low Level Image Features • Extraction Semantic Image Features • Building the Context Features • Experiments and Results • Improvements [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. [2] - Carson, C. , Belongie, S. , Greenspan, H. , and Mali, J. 1998. Color and texture-based image segmentation using EM and its application to context-based image retrieval, ICCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 2

INTRODUCTION [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. [2] - Carson, C. , Belongie, S. , Greenspan, H. , and Mali, J. 1998. Color and texture-based image segmentation using EM and its application to context-based image retrieval, ICCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 3

LOW LEVEL IMAGE FEATURES EXTRACTION OVERVIEW • Downsize the images to 60 × 80 pixels • Extract color information • Extract texture information • Extract global position information [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. [2] - Carson, C. , Belongie, S. , Greenspan, H. , and Mali, J. 1998. Color and texture-based image segmentation using EM and its application to context-based image retrieval, ICCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 4

LOW LEVEL IMAGE FEATURES EXTRACTION COLOR FEATURES EXTRACTION L* Component RGB to CIE L*a*b* a* Component RGB image b* Component [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. [2] - Carson, C. , Belongie, S. , Greenspan, H. , and Mali, J. 1998. Color and texture-based image segmentation using EM and its application to context-based image retrieval, ICCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 5

LOW LEVEL IMAGE FEATURES EXTRACTION COLOR FEATURES EXTRACTION RGB image L* Component a* Component b* Component [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. [2] - Carson, C. , Belongie, S. , Greenspan, H. , and Mali, J. 1998. Color and texture-based image segmentation using EM and its application to context-based image retrieval, ICCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 6

LOW LEVEL IMAGE FEATURES EXTRACTION Color Features [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. [2] - Carson, C. , Belongie, S. , Greenspan, H. , and Mali, J. 1998. Color and texture-based image segmentation using EM and its application to context-based image retrieval, ICCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 7

LOW LEVEL IMAGE FEATURES EXTRACTION TEXTURE FEATURES EXTRACTION Polarity Anisotropy RGB image Contrast [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. [2] - Carson, C. , Belongie, S. , Greenspan, H. , and Mali, J. 1998. Color and texture-based image segmentation using EM and its application to context-based image retrieval, ICCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 8

LOW LEVEL IMAGE FEATURES EXTRACTION Color Features Texture Features [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. [2] - Carson, C. , Belongie, S. , Greenspan, H. , and Mali, J. 1998. Color and texture-based image segmentation using EM and its application to context-based image retrieval, ICCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 9

LOW LEVEL IMAGE FEATURES EXTRACTION POSITION FEATURES EXTRACTION RGB image [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. [2] - Carson, C. , Belongie, S. , Greenspan, H. , and Mali, J. 1998. Color and texture-based image segmentation using EM and its application to context-based image retrieval, ICCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 10

LOW LEVEL IMAGE FEATURES EXTRACTION Color Features Texture Features Position Features [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. [2] - Carson, C. , Belongie, S. , Greenspan, H. , and Mali, J. 1998. Color and texture-based image segmentation using EM and its application to context-based image retrieval, ICCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 11

SEMANTICS FEATURES EXTRACTION • Semantic Layers used: - Building - Tree - Road - Sky • Example (for building): 1=building, 0=no building [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 12

SEMANTICS FEATURES EXTRACTION • In Test images: no ground truth → Use 4 SVM binary classifiers (input: low-level feature image) • Training set: 10 000 samples per category True Semantic Label Learned Semantic Label [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 13

SEMANTICS FEATURES EXTRACTION • ROC curve for the SVM classifiers [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 14

BUILDING THE CONTEXT FEATURES • Image has been converted to 20 layers: - 4 binary semantic features - 3 texture features - 3 color features - 10 global position features • Data sampled at 8 orientations and radii of 3, 5, 10, 15, 20 pixels • 40 samples: • (40)(20) = 800 dimensional context feature per pixel Green: size of car Red: size of pedestrian [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 15

EXPERIMENTS AND RESULTS Fidelity of semantic information Empirical semantic features: • Four SVMs • Four classes: building, tree, road, sky • Three features: colour, texture, position • Training and testing • Cross-validation and ROC [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 16

EXPERIMENTS AND RESULTS Performance of the context detector • Comparison with ROC curves of true high-level context detector and appearance detector • Appearance detector outperforms the context-based [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 17

EXPERIMENTS AND RESULTS Relative importance of context features • Comparison of four context classifiers • Low-level feature-based detection only marginally improved by addition of semantic features [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 18

EXPERIMENTS AND RESULTS Relative importance of context features • Testing of possible overlap of context with target object • Low-level and high-level classifiers at d ϵ{3, 5, 10, 15, 20} • Semantic features only important at farther distances [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 19

IMPROVING OBJECT DETECTION WITH CONTEXT Dataflow of the a rejection cascade • Tune thresholds THC and THA. • Different ROCs • Validation set of 200 images [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 20

IMPROVING OBJECT DETECTION WITH CONTEXT • Tuning the context threshold • Three different objects • Horizontal lines indicate the performance of the system with no context • The marks the selected parameters for the system • The ROCs of full system performance [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 21

CONCLUSIONS • An effective context detection system • Rejection cascade architecture • Importance of contextual cues • Good performance when the appearance information is weak (critically low resolution and very noisy images) • Ways of extracting context information [1] - Wolf, L. , and Bileschi, S. 2006. A critical view of context, IJCV. 21/11/2010 Critical View of Context – Alexandru Rusu, Guillaume Lemaître, Isabel Rodes and Oscar Ramos 22