Emerging Topics in Video Surveillance Rogerio Feris IBM

Emerging Topics in Video Surveillance Rogerio Feris IBM TJ Watson Research Center rsferis@us. ibm. com http: //rogerioferis. com 1 Video Surveillance E 6998 -007 Senior/Feris/Tian

Outline Ø Video Surveillance in Crowded Scenarios Ø Online Learning – Self-adaptation in Surveillance Ø Other Recent Topics 2 Video Surveillance E 6998 -007 Senior/Feris/Tian

Simple Scenarios Ø Few Objects – Background Subtraction + Tracking + Highlevel Event/Alert Detection Ø Current systems work well 3 Video Surveillance E 6998 -007 Senior/Feris/Tian

Crowded Scenarios Ø Many objects, occlusions, shadows, etc. Object Segmentation, Tracking and Event Analysis in crowded scenarios: Open Problem! 4 Video Surveillance E 6998 -007 Senior/Feris/Tian

Parts-based Detectors [Pedro et al, A discriminatively trained, multiscale, deformable part model, CVPR’ 08] Ø Occlusion Handling Ø Root filter (low-res) + Parts filters (high-res) 5 Video Surveillance E 6998 -007 Senior/Feris/Tian

Parts-based Detectors Ø Score of a window: score of root + score of parts Ø Score of Parts: Appearance + Geometry Ø Efficient localization of parts through Dynamic Programming Ø SVM Classification (Structured prediction) 6 Video Surveillance E 6998 -007 Senior/Feris/Tian

Detecting Pedestrians in Crowds [Leibe et al, Pedestrian Detection in Crowded Scenes, CVPR’ 05] Ø Combination of different models: bag of features, segmentation, and chamfer matching 7 Video Surveillance E 6998 -007 Senior/Feris/Tian

Tracking in Crowds [Andriluka et al, People-tracking-by-detection and people-detection-bytracking, CVPR’ 08] Ø Extends [Leibe et al, CVPR’ 05] to temporal-domain and person articulation (parts) estimation Ø Click for Video Demo 8 Video Surveillance E 6998 -007 Senior/Feris/Tian

Crowd Segmentation [Dong et al, Fast Crowd Segmentation Using Shape Indexing, ICCV’ 07] 9 Video Surveillance E 6998 -007 Senior/Feris/Tian

Crowd Analysis [Ali & Shah, A Lagrangian Particle Dynamics Approach for Crowd Flow Segmentation and Stability Analysis, CVPR’ 07] 10 Video Surveillance E 6998 -007 Senior/Feris/Tian

Online Learning 11 Video Surveillance E 6998 -007 Senior/Feris/Tian

Offline Adaboost Learning Ø Adaboost ensembles many weak classifiers into one single strong classifier § Initialize sample weights § For each cycle: Ø Find a classifier/rectangle feature that performs well on the weighted samples Ø Increase weights of misclassified examples § Return a weighted combination of classifiers 12 Video Surveillance E 6998 -007 Senior/Feris/Tian

Offline Adaboost Learning Major Problems: Ø Large number of examples required to train a robust classifier Ø time consuming to label data Ø slow training (may take several days) Ø No Adaptation to particular surveillance scenarios 13 Video Surveillance E 6998 -007 Senior/Feris/Tian

Learning from Small Sets Ø Choice of Features (Levi & Weiss, CVPR’ 04) Ø Co-Training (Levin & Viola, ICCV’ 2003) Online Adaptation: Ø Online Boosting (Oza’ 01, Javed’ 05, Bischof’ 06, Pham’ 07) 14 Video Surveillance E 6998 -007 Senior/Feris/Tian

Online Boosting [Oza, 2001] Ø Train a generic strong classifier (set of weak classifiers, # of weak classifiers fixed) on a small training set. ØOnline Process: Ø Given one single example with known label: Ø “Slide” the example over each weak classifier Ø When the weak classifier receives the example Ø update the weak classifier online Ø update the weight of the example and pass to the next weak classifier 15 Video Surveillance E 6998 -007 Senior/Feris/Tian

Online Boosting [Oza, 2001] 16 Video Surveillance E 6998 -007 Senior/Feris/Tian

Online Boosting [Oza, 2001] 17 Video Surveillance E 6998 -007 Senior/Feris/Tian

Online Boosting Car and People Detection [Omar Javed, CVPR’ 05] Ø Train a generic strong classifier (set of weak classifiers, # of weak classifiers fixed) on a small training set. Ø While running the classifier on unlabeled data, if an example is confidently predicted by a subset of weak classifiers use it for online learning Ø “Co-training framework” Ø BGS used for efficiency, for using more expensive features, and for balancing the number of positive and negative examples 18 Video Surveillance E 6998 -007 Senior/Feris/Tian

Online Boosting Car and People Detection [Omar Javed, CVPR’ 05] 19 Video Surveillance E 6998 -007 Senior/Feris/Tian

More Recent Work Ø [Helmut & Hurst, Online Boosting and Vision, CVPR’ 06] Ø [Bo Wu & Nevatia, Improving Part-based Object Detection by Unsupervised, Online Boosting, CVPR’ 07] Ø [Pham & Cham, Online Learning Asymmetric Boosted Classifiers for Object Detection, CVPR’ 07] Ø [Huang et al. , Incremental Learning of Boosted Face Detector, ICCV’ 07] – Boosting Adaptation Ø IEEE Online Learning for Classification Workshop (CVPR’ 08) 20 Video Surveillance E 6998 -007 Senior/Feris/Tian

Other Recent Topics 21 Video Surveillance E 6998 -007 Senior/Feris/Tian

High-Resolution Imagery [Kopf et al, Capturing and Viewing Gigapixel Images, SIGGRAPH’ 07] Ø How can we make use of high-resolution in video analytics? Ø Much more info – e. g. , in face reco: skin texture, iris, etc. 22 Video Surveillance E 6998 -007 Senior/Feris/Tian

Next Generation Neural Networks [Hinton, Reducing the dimensionality of data with neural networks, Science 2006] Ø New algorithm for learning deep belief nets Ø State-of-the art results in MNIST digit dataset (better than SVMs) Ø Youtube talk at Google: http: //www. youtube. com/watch? v=Ayz. OUbk. Uf 3 M Ø Matlab Code: http: //www. cs. toronto. edu/~hinton/ 23 Video Surveillance E 6998 -007 Senior/Feris/Tian

Learning with lots of data Ø How can we recognize thousands of products in a retail store for loss prevention? Ø 80 Million Tiny Images (http: //www. cs. nyu. edu/~fergus/) Surveillance with Moving Cameras Ø Cameras in vehicles, or even wearable cameras. New challenges: object detection, etc. Ø [Leibe et al, Dynamic 3 D Scene Analysis from a Moving Vehicle, CVPR 2007] 24 Video Surveillance E 6998 -007 Senior/Feris/Tian

Many more recent topics: Ø Check for papers in recent computer vision conferences (like CVPR, ICCV, and ECCV) and also specialized workshops/conferences such as AVSS and PETS 25 Video Surveillance E 6998 -007 Senior/Feris/Tian