Video Introduction Background Models Static Background Frame difference

Video ØIntroduction ØBackground Models Static Background, Frame difference, Running Average, Selectivity, Median, Running Gaussian Average, GMM o Shadow Detection o ØTracking o Exhaustive Search, Mean Shift, Optical Flow, Feature Point Tracking ØPerformance Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 1

Introduction – Objects of interest Reliably detecting moving objects of interest in a scene. 1. 2. 3. Motion detection Moving object detection & location Derivation of 3 D object properties When is an object of interest? Size Max and min velocity and acceleration Assumptions: Mutual correspondence Common motion Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 2

Introduction – Common problems Illumination & appearance changes Gradual (e. g. time of day) Sudden (e. g. clouds, lights) Shadows Weather (e. g. rain, snow) Background changes Objects becoming part of the background Objects leaving the background Background objects oscillating slightly Setup Camera motion Frame rate Field of view Distance to objects Location of camera Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 3

Introduction – Difference images Image subtraction. . Colour image version: Process each channel separately? Just process hue? absdiff( frame, background, difference ); © Reproduced by permission of Dr. James Ferryman, University of Reading False Positives False Negatives The threshold (ε) may be… Too low or too high Just right? High contrast Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 4

Background Models – Static Background | fn(i, j) - b(i, j) | > T Simplest approach Sensitive to the Threshold T absdiff( current_frame, first_frame, difference ); cvt. Color( difference, moving_points, CV_BGR 2 GRAY ); threshold( moving_points, 30, 255, THRESH_BINARY ); Mat display_image = Mat: : zeros( moving_points. size(), CV_8 UC 3 ); current_frame. copy. To( display_image, moving_points ); Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 5

Background Models – Static Background | fn(i, j) - b(i, j) | > T Background b(i, j) = first frame? © Reproduced by permission of Dr. James Ferryman, University of Reading Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 6

Background Models – Frame Difference | fn(i, j) – fn-1(i, j) | > Threshold Depends on the speed of the objects and the frame rate Sensitive to the Threshold © Reproduced by permission of Dr. James Ferryman, University of Reading Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 7

Background Models – Running Average Adapts to changing scene Incorporates moving objects… © Reproduced by permission of Dr. James Ferryman, University of Reading accumulate. Weighted( current_frame_gray, running_average_background, 0. 01 ); Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 8

Background Models – Selectivity (with running average) Avoids incorporating moving objects Adapts to scene changes? © Reproduced by permission of Dr. James Ferryman, University of Reading accumulate. Weighted( current_frame_gray, running_average_background, 0. 01, foreground_mask ); Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 9

Background Models – Colour Running Average Selectivity (with running average) © Reproduced by permission of Dr. James Ferryman, University of Reading Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 10

Background Models – Colour vector<Mat> input_channels(3); split(current_frame, input_channels); vector<Mat> channels(3); background. convert. To(temp_background, CV_8 U); absdiff(temp_background, current_frame, difference); split(difference, channels); temp_sum = (channels[0] + channels[1] + channels[2])/3; threshold(temp_sum, foreground, 30, 255, THRESH_BINARY_INV); split(background, channels); accumulate. Weighted(input_channels[0], alpha, foreground); accumulate. Weighted(input_channels[1], alpha, foreground); accumulate. Weighted(input_channels[2], alpha, foreground); invert. Image(foreground, foreground); Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 11

Background Models – Colour if (update_foregrond) { accumulate. Weighted(input_channels[0], alpha /3. 0, foreground); accumulate. Weighted(input_channels[1], alpha /3. 0, foreground); accumulate. Weighted(input_channels[2], alpha /3. 0, foreground); } merge(channels, background); if (clean_binary_images) { morphology. Ex(foreground, closed_image, MORPH_CLOSE, Mat()); morphology. Ex(closed_image, f oreground, MORPH_OPEN, Mat()); } foreground_image. set. To(Scalar(0, 0, 0)); current_frame. copy. To(foreground_image, foreground); Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 12

Background Models – Selectivity (with running average) Reduce the incorporation of moving objects © Reproduced by permission of Dr. James Ferryman, University of Reading Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 13

Background Models – Median: Middle value (from an ordered list) No. of frames (m) Histogram quantisation? Computational expense Adding, storing and removing frames Change in median can be tracked inexpensively from frame to frame Can be approximated using aging Can also use selective update Could use the Mode instead… (Most common value) Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 14

Background Models – Median Results Learning Rate = 1. 001 Learning Rate = 1. 005 Learning Rate = 1. 02 © Original Video reproduced by permission of Dr. James Ferryman, University of Reading Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 15

Background Models – Median Algorithm Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 16

Background Models – Running Gaussian Average Fit a Gaussian distribution over the histogram Background PDF (μ, σ) Update using running average Foreground test Threshold is a multiple of the standard deviation Can also use selective update e. g. Pfinder (Wren et al. 1997) Problems? Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 17

Background Models – Gaussian Mixture Model How to deal with multi-modal background pixels e. g. from trees, water Stauffer & Grimson, 2000 Algorithm presented is based on that in Sonka (3 rd edition) pp. 777 -780 Model multiple values (3 -5) at each point. Unsupervised learning… Most popular method for background modelling © Reproduced by permission of Dr. James Ferryman, University of Reading Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 18

Background Models – GMM Model details Model each pixel fn(i, j) using k Gaussian distributions Nk = N(μk, Σk) πn μn Σn Covariance between two channels a and b for pixel (i, j) at frame t: Σt ((X t(i, j, a) - μk(i, j, a)) * ((Xt(i, j, b) - μk(i, j, b)) Approximate using σ2 k Assume different components independent and of equal variance Set a learning constant α in the range 0. 01 … 0. 1 Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 19

Background Models – GMM Model Update For a new sample fn(i, j) Select the best close Gaussian distribution close = within 2. 5 σn (i, j, m) of μn (i, j, m) If there is a best close Gaussian l If there is no close Gaussian (replace one…) x = argmink(πn(i, j, k)) μn+1(i, j, x) = fn(i, j) σ 2 n+1(i, j, x) = 2. maxk σ 2 n(i, j, k) πn+1(i, j, x) = ½. mink πn(i, j, k) Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 20

Background Models – GMM Moving Points Identifying background distributions Define T, a proportion of the frames in which background pixels should be visible. Order the Gaussians by πn+1(i, j, k) / σ n+1(i, j, k) Gaussians 1. . B are considered background where B = argminb( (Σb=1. . k πn+1(i, j, k) ) > T ) Just check if best close Gaussian (or the new Gaussian distribution) is a background distribution Finally use morphological dilations and erosions to remove small regions and fill in holes. © Reproduced by permission of Dr. James Ferryman, University of Reading Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 21

Background Models – Shadow Detection Compare current frame with background image… (Prati 2003) Intensity / luminance / value drops Intensity / luminance / value has not dropped too much (below λ) Saturation does not increase too much Hue does not change too much Hue unpredictable & Change can decrease a little (Tattersall 2003) Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 22

Background Models – Shadow Detection © Reproduced by permission of Dr. James Ferryman, University of Reading Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 23

Background Models – Shadow Detection – Code Background. Subtractor. MOG 2 gmm; gmm( current_frame, foreground_mask ); threshold( foreground_mask, moving_points, 150, 255, THRESH_BINARY ); … threshold( foreground_mask, changing_points, 50, 255, THRESH_BINARY ); absdiff( moving_points, changing_points, shadow_points ); … Mat mean_background_image; gmm. get. Background. Image( mean_background_image ); Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 24

Tracking – Introduction Used in video surveillance, sports video analysis, vehicle guidance systems, etc. A hard task because objects may be undergoing complex motion may change shape may be occluded may change appearance due to lighting/weather may physically change appearance Approaches considered: Exhaustive search Mean Shift Optical Flow Feature based tracking Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 25

Tracking – Exhaustive Search Extract object to be tracked from frame Compare in all possible positions in future frame(s) Use a similarity metric E. g. normalised cross correlation Pick the best match Need extra degrees of freedom for scale and orientation. May fail if object motion is too complex. Template matching and chamfer matching support this type of tracking. Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 26

Tracking – Mean Shift Back-projects a histogram of the object into the current frame Searches for a region of the same size within the back projection looking for the highest (weighted) sum. Uses hill climbing to iteratively look for the (local) maximum Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 27

Tracking – Mean Shift Rect position(starting_position); Term. Criteria criteria( cv: : Term. Criteria: : MAX_ITER, 5, 0. 01); mean. Shift( back_projection_probabilities, position, criteria); Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 28

Tracking – Dense Optical Flow Compute a motion field (known as optical flow) for the entire image Direction & Magnitude © Reproduced by permission of Dr. James Ferryman, University of Reading Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 29

Tracking – Dense Optical Flow Based on the brightness constancy constraint Object points will have the same brightness over a short period of time Need to find the displacement minimise the residual error Video which will Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 30

Tracking – Dense Optical Flow To compute the optical flow the displacement is small… , assuming that Hence (given our previous equation So And reorganising Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 31

Tracking – Dense Optical Flow – Code calc. Optical. Flow. Farneback (previous_gray_frame, optical_flow, 0. 5, 3, 15, 3, 5, 1. 2, 0); cvt. Color (previous_gray_frame, display, CV_GRAY 2 BGR); for (int row = 4; row < display. rows; row+=8) for(int column = 4; column < display. cols; column+8) { Point 2 f& flow = optical_flow. at<Point 2 f>(row, column); line (display, Point(column, row), Point( cv. Round(column+flow. x), cv. Round(row+flow. y)), passed_colour); } gmm. get. Background. Image( mean_background_image ); Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 32

Tracking – Feature-based Optical Flow We cannot accurately compute optical flow for constant regions or along edges. Often better to compute optical flow just for features… (e. g. Lucas Kanade feature tracker)… © Reproduced by permission of Dr. James Ferryman, University of Reading Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 33

Performance – Ground Truth Formats 1. Label pixel masks • Label all pixels corresponding to objects of interest (e. g. moving people). • Very time consuming. • Very hard to get agreement particularly near boundaries. 2. Bounding Boxes • Put a box around all objects of interest. • Much faster to do. • Much more inaccurate (e. g. size of bounding box vs. area of object). 3. Labelled events • Simply label a frame or range of frames to indicate when some event has taken place (e. g. car has stopped). • Much easier to compute. • May need a bounding box or labelled pixels as well for the frame. • Quite hard to agree on what frame an event occurs in. Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 34

Performance – Metrics For labelled pixel masks or labelled events we can use metrics like Precision, Recall, F 1 measure, etc. For bounding boxes we must use other metrics such as 1. Dice coefficient 2. Overlap 3. Percentage of Lost Tracks • Lost Track = Overlap falls below a threshold • Threshold = 10% or 20% ? ? ? Video Based on A Practical Introduction to Computer Vision with Open. CV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014 Slide 35