people Detection Ali Taalimi Prof Abidi 11192012 Outline

people Detection Ali Taalimi Prof. Abidi 11/19/2012

Outline • Object Detection – Macro Scheme – Main Component • People Detection • Crowd • Suggested Approach 3/1/2021 Slide 2/90

Macro Scheme Input Frames Object Detection • Scene Understanding • Scene Modeling Object Tracking 3/1/2021 Slide 3/90 • Temporal Coherency

Macro Scheme Bottom–Up Scheme Frames Patch List Patch Detector Identified Object List Patch Classifier Training class specific Patch model 3/1/2021 Slide 4/90 Grouping Tracking

Macro Scheme Top–Down Scheme Mask Frames Foreground Estimation Background Model (history) 3/1/2021 Slide 5/90 Silhouette List Grouping (Conn Comp) Classifier Fixed Rules Or Training Object List Tracking Trajectories

Main Components • method to extract relevant information from an image area occupied by a target. • representation for appearance and shape of a target to be used by the tracker and descriptive enough to cope with clutter. • links different instances of the same object over time which has to compensate for occlusion, clutter, illumination. 3/1/2021 Slide 6/90

Features • Low Level Features: Color: RGB, CIELUV, HIS Gradient: Local intensity changes within (different reflectance properties of object parts, skin, hair) and at the boundary (different reflectance properties of the object with background). Laplacian of Gaussian (Lo. G)/Difference of Gaussains (Do. G) Motion: to detect and localize objects over time (optical flow). Low level features cannot describe image contents completely. • Mid-Level Features: Using subset of pixels that represent structures (edges, interest points/regions) Interest points detector: select highly distinctive features that can be localized across multiple frames when pose and illumination change (corners). • High Level Features: Detect object as a whole based on its appearance Group mid level features background modeling Object Modeling (color based segmentation of the face detection) 3/1/2021 Slide 7/90

Target Representation • How we can define a target in terms of its shape and appearance. • A target representation is a model of the object that is used by a tracking algorithm using shape and appearance of the target. • Shape/Appearance info can be encoded at different levels of resolution. • Example: ü using bounding box or a deformable contour to approximate the shape of the target, ü using pdf of some appearance features computed within the target area for encoding appearance info. • Uncertainty factors, like illumination changes, clutter, target interaction and occlusion should be account for defining how to represent target. 3/1/2021 Slide 8/90

Target Representation Shape Representation: Basic, Articulated, Deformable Representations A. Yilmaz, O. Javed, and M. Shah, “Object tracking: A survey, ” ACM Comput. Surv. , vol. 38, no. 4, pp. 1– 45, 2006. 3/1/2021 Slide 9/90

Target Representation Shape Representation: Basic Models: 1) Point Approximation: • limitation: target occlusion (as they don’t provide an estimate of the size of target ) 2) Area Approximation: bound target with a rectangle or ellipse. Motion, color, gradient of the entire target area used for tracking. • Tracking is performed by estimation parameters describing possible transformations of center, axes of these approximated shapes. • Lack of depth 3) Volume Approximation: multiple object occluding can be handled using spatial volume occupied by the target. • Lack of generality: only for objects available in training dataset. 4) Articulated models: approximate shape of the target by combining set of rigid models based on topological connections and motion constraints • e. g. full body tracking of human using topology of human skeleton 3/1/2021 Slide 10/90

Target Representation Shape Representation: Deformable Models: Shape rigidity or kinematic assumptions doesn’t work for all target classes. Where it happens? • Prior information on the object shape may not be available. • Object have deformations that are not well modeled by canonical joints. 1) Fluid Models: Instead of tracking the entire target area, interest points can be identified on the object and then used to track its parts. • No explicit motion constraints between the different parts • Example: Tracking detectable corners of the object. • Stable object tracking in occluded state. • Problem: how to group the points (which of them belong to the same object? ). 2) Contours: more accurate description of the target. • Contour based trackers use set of control points positioned along the contour. • Concatenation of the coordinates of control points will be considered. • Use prior info of the target shape 3/1/2021 Slide 11/90

Part Based Model • model consists of a global “root” filter and several part models. • Both root and part filters are computed by dot product between a set of weights and HOG features. two different scales: 1) Coarse features are captured by a rigid template of entire detection window. 2) Finer scale features are captured by part templates with respect to the detection window. P. Felzenszwalb. A discriminatively trained, multiscale, deformable part model. In CVPR, 2008. 3/1/2021 Slide 12/90

Target Representation Appearance Representation • The appearance representation is a model of the expected projection of the object appearance onto the image plane. • Unlike before, the representation of the appearance of the target may be specific to a single object and does not generalize across objects of the same class. • Usually paired with a function that given the image estimate the likelihood of an object being in a particular state. Target Position Input image Feature Extraction Color, Gradient Appearance Representation, Template, Histogram Learning Target Modeling Template: encode positional information of color values of all pixels within the target area. Histogram: of Color, of Gradient 3/1/2021 Slide 13/90

Outline • Object Detection • People Detection – ROI Selection – Background Modeling – Classification – Tracking • Crowd • Suggested Approach 3/1/2021 Slide 14/90

Challenges Jitendra Malik 3/1/2021 Slide 15

People Detection • main components of a people detection system: ü ü hypothesis generation (ROI selection) classification (model matching, verification ) Tracking (temporal integration ) Evaluation • ROI selection: initialized by general low-level features or prior scene knowledge. • Classification/Tracking require models of the people class, in terms of geometry, appearance or dynamics. New Images Target Class Samples Learning Class Model Nontarget Class Samples 3/1/2021 Slide 16/90 Classification detected candidate targets

ROI Selection sliding window technique: detector windows at various scales and locations are shifted over the image. ü computational costs are often too high. ü speedups by either coupling with a classifier cascade, or by restricting the search space based on prior information about the target object class: Ø such as geometry of peoples, e. g. , object height or aspect ratio features derived from the image data ü object motion ü Background subtraction ü interest-point detectors (ISM) confidence density of the detector: ü sliding-window: density is implicitly sampled in a discrete 3 D grid (location and scale) by evaluating the different detection windows with a classifier. ü Feature based: density is explicitly created in a bottom-up fashion through probabilistic votes cast by matching, local features. 3/1/2021 Slide 17/90

Classification • Receives a list of ROIs that are likely to contain a people. • verification (classification) will work on people appearance models, using various spatial and temporal cues. • a given image/subregion is assigned to either the people or nonpeople class, based on its class posterior probabilities. • class posterior probabilities: probability of having people in that region given a model. • how to estimate posterior probability: ü generative and discriminative models • Challenge: ü missing detections: not all people are detected in each frame ü false positive detections: detect non people ü in case of no depth or scene information (e. g. , ground plane), the detector does not know where to expect objects of which size in the image. 3/1/2021 Slide 18/90

Classification D. Geronimo, “Survey on people detection for advanced driver assistance systems, ”IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 32, no. 7, pp. 1239– 1258, 2010. 3/1/2021 Slide 19/90

Classification Generative Models: modeling the appearance of the people class in terms of its class conditional density function. Shape models: reducing variations in people appearance due to lighting or clothing. 1)Discrete approaches: represent the shape manifold by a set of large amount of exemplar shapes. 2)Continuous shape models: parametric representation of the class- conditional density, learned from a set of training shapes. 3)Combined shape and texture models: combine shape and texture information within a compound parametric appearance model. ü texture cue represents the variation of the intensity pattern across the image region of target objects. ü separate statistical models for shape and intensity variations. 3/1/2021 Slide 20/90

Classification Discriminative Models: approximate the Bayesian MAP decision by learning the parameters of a discriminant function (decision boundary) between the people and nonpeople classes from training examples. Features: 1)Haar wavelet feature 2)codebook feature patches 3)gradient orientation histograms: dense and sparse (SIFT) 4)spatial configuration of salient edge-like structures: shapelet, edgelet 5)spatiotemporal features to capture human motion, especially gait Classifier Architectures: determine an optimal decision boundary between pattern classes in a feature space. 1)Feed-forward multilayer neural networks 2)Linear/Nonlinear Support Vector Machines (SVMs) 3)Ada. Boost 3/1/2021 Slide 21/90

Classification Multipart representations: • break down the complex appearance of the people class into subparts. • make local pose-specific people clusters, and training of a specialized expert classifier for each subspace. • integrate the individual expert responses to a final decision (model for geometric relations between parts) M. Enzweiler, “Multi-cue ped. classification with partial occlusion handling, ” in IEEE Conf. Computer Vision and Pattern Recognition, 2010. 3/1/2021 Slide 22/90

Classification Multipart representations: codebook representations: represent peoples in a bottom-up fashion as assemblies of local codebook features, combined with top-down verification. Codebook Generation B. Leibe, E. Seemann, B. Schiele, people detection in crowded scenes, CVPR 2005. people Detection in Crowded Street Scenes, Seemann et al, CVPR 2005. 3/1/2021 Slide 23/90

Learning Algorithms Different Learning Algorithms/Classifiers Definition Properties SVM Find a decision boundary by maximizing the margin between the different classes. Decision boundary can be linear. Data can be of any type, i. e. scalar, vector features, intensity Ada. Boost Constructs a strong classifier by attaching weak classifiers in an iterative greedy manner. High speed because of cascades. Can be combined with any classifier to find weak rules Neural Networks Different layers of neurons provide a nonlinear decision. Many configurations and parameters to choose. Raw data (intensity, gradient mag) is often used. 3/1/2021 Slide 24/90

Result of Classification • Holistic algorithms are unable to deal with high variability: ü nonstandard poses greatly affect their performance ü diversity of poses causes many peoples to be poorly represented during training (e. g. , running people, children, etc. ) • Parts-based algorithms that rely on dynamic part detection, handle pose changes better than holistic approaches. • Support vector machines (SVMs) and boosting are the most popular choices for LEARNING/Classifier. • Nearly all modern detectors employ: ü ü 3/1/2021 Slide 25/90 gradient histograms grayscale (e. g. Haar wavelets ) color, texture self-similarity and motion features.

Overall Performance Only Detection without Tracking (1) peoples at least 80 pixels tall, 20 -30% of all peoples are missed (at 1 false alarm per 10 images). (2) Performance degrades catastrophically for smaller peoples. While peoples 30 -80 pixels tall, around 80% are missed by the best detectors (at 1 false alarm per 10 images). (3) Performance degrades similarly under partial occlusion (under 35% occluded) (4) Performance is very bad at far scales (under 30 pixels) and under heavy occlusion (over 35% occluded). Under these conditions nearly all peoples are missed even at high false positive rates. P. Dollar, C. Wojek, B. Schiele, and P. Perona. people detection: An evaluation of the state of the art. In PAMI, volume 99, 2012 3/1/2021 Slide 26/90

Background Modeling Statistical Background Modeling (GMM, KDE) Background Clustering Neural Network Background Modeling Background Estimation steps and issues: übackground modeling übackground initialization übackground maintenance üforeground detection üchoice of the feature size (pixel, a block or a cluster) üchoice of the feature type (color/edge/motion/texture features) critical situations in video sequence: Bootstrapping, Camouflage, Moved background objects, Inserted background, Waking foreground object, Sleeping foreground object and Shadows, Dynamic backgrounds, Illumination changes two constraints: less computation time, less memory requirement 3/1/2021 Slide 27/90

Statistical Background Modeling Category Methods First Category Mixture of Gaussians (MOG) Kernel Density Estimation (KDE) Principal Components Analysis (PCA) Second Category Support Vector Machine (SVM) Support Vector Regression (SVR) Support Vector Data Description (SVDD) Third Category Single General Gaussian (SGG) Mixture of General Gaussians (MOGG) Independent Component Analysis (ICA) Incremental Non Negative Matrix Factorization (INMF) Incremental Rank-(R 1, R 2, R 3) Tensor (IRT) T. Bouwmans, F. El Baf, and V. B. Statistical Background Modeling for Foreground Detection: A Survey, volume 4 of Handbook of Pattern Recognition and Computer Vision, chapter 3. World Scientific Publishing, 2010. 3/1/2021 Slide 28/90

Disadvantages of MOG • the number of Gaussians must be predetermined • the need for good initializations • the dependence of the results on the true distribution which can be non-Gaussian • slow recovery from failures • the need for a series of training frames absent of moving objects • the amount of memory required in this step. Solution: Intrinsic, Extrinsic 3/1/2021 Slide 29/90

MOG improvements Intrinsic Background Step Parameters Background Initialization Variable K Variables μ , σ , ω Methods Markov Random Fields Background Maintenance Variable K Variables μ , σ , ω Learning rates α , ρ Foreground Detection Different measure for the matching test Probabilities Foreground model 3/1/2021 Slide 30/90 Extrinsic Hierarchical approaches Multi-level approaches Multiple backgrounds Graph cuts Multi-layer approaches Tracking feedback

Features Improvements of the MOG Original method uses only RGB values of a pixel without assuming spatial knowledge. Feature Size Block Cluster Feature Type Color features Normalized RGB YUVHSV HSI Luv Improved HLS YCr. Cb Edge features Texture features Spatial features Motion features HOG features Video features 3/1/2021 Slide 31/90

Subspace Learning using PCA • Subspace learning offer a good framework to deal with illumination changes as it allows taking into account spatial information. • By assuming that the large part of image is background, expect to have only background using M largest Eigen vectors. Limitation: • size of the foreground object must be small • Foreground should not appear in the same location during a long period in the training (stationary/slow motion foreground) • time consuming specially for colorful image. 3/1/2021 Slide 32/90

Tracking provide correspondences between the regions of consecutive frames based on the features and a dynamic model. Challenges in video tracking: üRobustness to clutter, Occlusion üFalse positives/negatives üStability Target changes its pose so its appearance as seen by camera 3/1/2021 Slide 33/90 occlusion by another moving object Clutter in video tracking: Objects in the background share similar color/shape with target

Tracking Challenges Similarity of appearance (clutter) Tracking Challenges Illumination Scene Variations of appearance Rotation Translation Deformation 3/1/2021 Slide 34/90 Object Pose Weather Partial Occlusions Sensor noise Total

Tracking 1) Prevent false detections using all of the data within a spatial/temporal sliding window representing the most recent part of video. 2) Consider motion estimates to have correct data associations. 3) Predicting future peoples positions, thus feeding the foreground segmentation algorithm with pre-candidates. Video Tracking In Computer Vision: ü Window Tracking ü Feature Tracking: detectable part of images v Tracking Local Features (corners, edges, lines) v Optic Flow Methods v Tracking Extended Features (ellipses, rectangles, contours, regions) ü Tracking Deformable Contours (snake, parametric geometric model, deformable templates) ü Visual Learning: instead of capturing shape/appearance apriori, learn it by examples. 3/1/2021 Slide 35/90

Tracking systems address: motion and matching ü Motion problem: identify a limited search region in which the element is expected to be found with high probability v simplest way: define the search area in the next frame as a fixed-size region surrounding the target position in the previous frame. v Kalman filter (KF), Particle Filtering, Spatio-Temporal MRF, Graph correspondence, Event cones ü Matching problem: (also known as detection or location) identify the image element in the next frame within the designated search region v similarity metric to compare candidates in the previous and current frame. v data association: occurs in the presence of interfering target/trajectories 3/1/2021 Slide 36/90

Tracking by Detection Approach • involve the continuous application of a detection algorithm in individual frames and the association of detections across frames. • generally robust to changing background and moving cameras. • Why is association between detections and targets difficult? ü Detection result degrades in occluded scene. ü Detector output is unreliable and sparse, i. e. , detectors only deliver a discrete set of responses and usually yield false positives and missing detections. The output of a person detector (right: ISM , left: HOG) with false positives and missing detections. M. D. Breitenstein, et al, Online multi-person tracking-by-detection from a single, uncalibrated camera, " IEEE Trans. on Pattern Analysis and Machine Intell. (PAMI), vol. 33, no. 9, 2011. 3/1/2021 Slide 37/90

Outline • Object Detection • People Detection • Crowd – – Crowd Challenges Crowd Information Extraction Crowd Dynamic/Analysis Tracking in Crowd Scene • Suggested Approach 3/1/2021 Slide 38/90

Different Levels of Crowd Densities (a) Very low density; (b) Low density; (c)Moderate density; (d) High density; (e) Very high density A. Marana, L. da Costa, R. Lotufo, and S. Velastin, “On the efficacy of texture analysis for crowd monitoring, ” in Proc. Int. Symp. Computer Graphics, Image Processing, and Vision (SIBGRAPI’ 98), Washington, DC, 1998, p. 354. 3/1/2021 Slide 39/90

Crowd Challenges • straightforward extension of techniques designed for noncrowded scenes don’t work for crowded situations. • Because of severe occlusion, it is difficult to segment and track each individual in crowd. • In high-density video sequences the accuracy of traditional methods for object tracking decreases as the density of people increases. • Dynamic of a crowd itself is complex. ü Goal Directed ü Psychological Characteristic • Occlusion Reasoning: ü If a partially occluded person is detected and associated to a trajectory, the classifier will be updated with noise and performance will degrade. 3/1/2021 Slide 40/90

Crowd information extraction 1. Crowd density measurement: crowd density or counting number of peoples 2. Recognition: In extremely cluttered scenes, individual people segmentation is exceedingly difficult. ü Face and Head Recognition ü people and Crowd recognition: v Occlusion handling v Moving Camera: on-board vision system to assist a driver v Spatial-temporal methods ü Tracking v Solve occlusion during and after the occurrence of occlusions. v using traceable image features v Human body model: models human body parts. Tracking is implemented by probabilistic data association, i. e. matching the object hypotheses with the detected response v Tracking inference strategies: particle filtering, MHT, JPDAF, Hungarian algorithm, greedy search, … 3/1/2021 Slide 41/90

Crowd Analysis Crowd analysis using computer vision techniques, Jacquez et al, 2010, IEEE signal processing magazine 3/1/2021 Slide 42/90

Crowd Dynamics/Analysis CROWD DYNAMICS Crowds can be characterized considering three different aspects: 1. image space domain 2. sociological domain: study the behavior of people in several years. 3. computer graphics domain. CROWD ANALYSIS Three issues of analysis of crowded scenes: 1. people counting/density estimation models 2. tracking in crowded scenes 3. crowd behavior understanding models counting vs. tracking: Similarity: the goal of both is to identify the participants of a crowd Difference: counting is about to estimate of the number of people, position and temporal evolution isn't considered. Tracking determines the position of each person in the scene as a function of time. 3/1/2021 Slide 43/90

Crowd Analysis PEOPLE COUNTING/DENSITY ESTIMATION MODELS • pixel-based analysis: ü based on very local features (individual pixel analysis via background subtraction or edge detection). ü mostly focused on density estimation rather than precise people counting. • texture-based analysis: ü requires the analysis of image patches. ü explores higher-level features comparing to pixel-based approaches • object-level analysis: ü try to identify individual objects in a scene. ü produce more accurate results comparing to the other two ü identifying individuals is feasible in low density crowds ü Very hard to solve for denser crowds ü Highly dependent on the extraction of foreground blobs that generate the image features 3/1/2021 Slide 44/90

Tracking In Crowded Scenes unstructured environments: ümotion of a crowd appears to be random with different participants moving in different directions over time (e. g. , a crossway). üApproach should allow each location of the scene to have various crowd behaviors. üusing the head as the point of reference rather than the entire body. because heads are rarely obscured from overhead surveillance cameras and are generally not obscured by clothing. üMultiple Target Tracking: v appearance based methods: 1) feed-forward systems which use only current and past observations to estimate the current state. 2) data association based methods which use future information also, to estimate the current state, allowing ambiguities to be more easily resolved at the cost of increased latency 3/1/2021 Slide 45/90

Outline • Object Detection and Tracking • People Detection • Crowd • Suggested Approach 3/1/2021 Slide 46/90

Goal • • Make Reliable Detection: low FP, FN (miss detection) Segment multiple possibly occluded humans in the image. Detect head/face of each person in video sequences. To obtain consistent trajectories of multiple possibly occluding humans in the video sequence. • To track human robustly. • Occlusion Reasoning 3/1/2021 Slide 47/90

My Main Scheme Regions of interest Preprocessing Foreground Segmentation Labeled ROIs Object Classification Verified and refined ROIs Verification/ Refinement Tracking

Detection Object detection can be performed by modeling and then classifying background and foreground. 1) Training classifier on the appearance of the background pixels and then a detection is associated with each connected region/blob of the foreground pixels. 2) Train a set of classifiers to encode the people (foreground). 3) Statistical shape-and-texture appearance models to define a representation of people appearance. Using multiple detection modules: • Shape based detection + texture based classification. • We should explain why certain modules were selected and how they were integrated in the overall system. 3/1/2021 Slide 49/90

Detection/Tracking Head Detection for Tracking: • Detecting head in specific and part based model in general can be more reliable rarely obscured from cameras and clothing. • locating faces directly may not be possible due to occlusion, pose variations, or a relatively small size of the face region compared to the whole image. Make Robust Multi-Target Tracker: • Instead of using detection and classification results to guide the tracker, we can use Coupling Detection and Tracking Using Tracklet: ü update tracking model by the detection confidence density. ü Using Spatio/Temporal knowledge like motion tracker and object attributes. • Example: Detection fails in occluded situation, tracker may help. In case of abrupt motion changes tracker drifts, detection can be helpful. 3/1/2021 Slide 50/90

References • • • • 2006_A survey of advances in vision-based human motion capture and analysis 2006_Object Tracking_ a survey 2006_Video Tracking_ A Concise Survey 2006_Tracking People in Crowded Scenes 2008_Machine Recognition of Human Activities_ A Survey 2008_Crowd analysis- A survey 2008_people Tracking by Associating Tracklets using Detection Residuals 2008_Robust Object Tracking by Hierarchical Association of Detection Responses 2009_Monocular people detection_ Survey and experiments 2010_survey people Detection for Advanced Driver Assistance Systems 2010_Crowd analysis using computer vision techniques 2011_A Review of Computer Vision Techniques for the Analysis of Urban Traffic 2011_Stable Multi-Target Tracking in Real-Time Surveillance Video 2011_Online Multiperson Tracking-by-Detection from a Single, Uncalibrated Camera 2012_people Detection An Evaluation of the State of the Art 2013_Monocular Visual Scene Understanding Multi-Object Traffic Scenes 3/1/2021 Slide 51/90