Motion Tracking Image Processing and Computer Vision 8

Motion Tracking Image Processing and Computer Vision: 8

Introduction n Finding how objects have moved in an image sequence n n n Movement in space Movement in image plane Camera options n n Static camera, moving objects Moving camera, moving objects Image Processing and Computer Vision: 8 2

Contents n Acquiring targets n n n Image differencing Moving edge detector Following targets n n n Matching Minimum path curvature Model based methods Kalman filtering Condensation Hidden Markov Model Image Processing and Computer Vision: 8 3

Camera calibration revisited n Image to camera co-ordinate transformation n n Intrinsic parameters Camera to world co-ordinate transformation n Extrinsic parameters Image Processing and Computer Vision: 8 4

Camera to Image Co-ordinates Distortionless Camera n If n no distortions uniform sampling Co-ordinates linearly related n offset and scale Image Processing and Computer Vision: 8 5

Camera to Image Co-ordinates Distorting Camera n Periphery is distorted n k 2 = 0 is good enough Image Processing and Computer Vision: 8 6

Pinhole Camera f Z Optical centre Object Image and centre, object and centre are similar triangles. Image Processing and Computer Vision: 8 7

Camera and World Coordinates translate and rotate World frame ra e am e m a fr C Image Processing and Computer Vision: 8 8

System architecture Start Acquire target End Follow target Image Processing and Computer Vision: 8 9

Target acquisition n Finding a target to follow Differencing Moving edge detector Image Processing and Computer Vision: 8 10

Change and Moving Object Detection n n Simplest method of detecting change Compute differences between n n Live and background images Adjacent images in a sequence Image Processing and Computer Vision: 8 11

Image Differencing n Differences due to n n Moving object overlying static background Moving object overlying another moving object Moving object overlying same moving object Random fluctuation of image data Image Processing and Computer Vision: 8 12

Image Processing and Computer Vision: 8 13

Demo n n Inter frame differencing Difference from a background n p. Finder Image Processing and Computer Vision: 8 14

Background image n n n Detecting true differences required an accurate background Lighting changes? Camera movement? Image Processing and Computer Vision: 8 15

Background image updates n Periodically modify whole background n n Will include changes in new background Systematically incorporate non-changed portions of image into background Image Processing and Computer Vision: 8 16

Critique n Can identify changes in the image data n n But what do the changes mean? Need a second layer of processing n n To recognize changes Optical flow sidesteps this problem. . . Image Processing and Computer Vision: 8 17

Target following n n Observing the positions of an object or objects in a time sequence of images. Object matching Minimum path curvature Model based methods Image Processing and Computer Vision: 8 18

Matching n n n Locate objects in each image Match objects between images Use methods of previous lectures Image Processing and Computer Vision: 8 19

Minimum path curvature n Observations of two objects in three images Image Processing and Computer Vision: 8 20

Image Processing and Computer Vision: 8 21

Image Processing and Computer Vision: 8 22

Image Processing and Computer Vision: 8 23

Image Processing and Computer Vision: 8 24

Which is “best” solution? n One with overall straightest paths For each solution For each path Compute total curvature Sum Image Processing and Computer Vision: 8 25

Model based tracking n Mathematical model of objects’ motions: n n position, velocity (speed, direction), acceleration Can predict objects’ positions Image Processing and Computer Vision: 8 26

System overview Motion Model Predict Location Update? Verify Location Image Processing and Computer Vision: 8 27

Simple Motion Model n n Newton’s laws s = position u = velocity a = acceleration n n all vector quantities measured in image co-ordinates Image Processing and Computer Vision: 8 28

Prediction n Can predict position at time t knowing n n Position Velocity Acceleration At t=0 Image Processing and Computer Vision: 8 29

Uncertainty n n If some error in a - Da or u - Du Then error in predicted position - Ds Image Processing and Computer Vision: 8 30

Verification n Is the object at the predicted location? n Matching n n How to decide if object is found Search area n Where to look for object Image Processing and Computer Vision: 8 31

Object Matching n Compare n n n A small bitmap derived from the object vs. Small regions of the image Matching? n Measure differences Image Processing and Computer Vision: 8 32

Search Area: Why? and Where? n Uncertainty in knowledge of model parameters n n n Limited accuracy of measurement Values might change between measurements Define an area in which object could be n Centred on predicted location, s Ds Image Processing and Computer Vision: 8 33

Update the Model? n n Is the object at the predicted location? Yes n n No change to model No n n Model needs updating Kalman filter is a solution Image Processing and Computer Vision: 8 34

Kalman filter Mathematically rigorous methods of using uncertain measurements to update a model Image Processing and Computer Vision: 8 35

Kalman filter notation n Relates n Measurements y[k] n n System state x[k] n n n Relates system state to measurements Evolution matrix A[k] n n Position, velocity of object, etc Observation matrix H[k] n n e. g. positions Relates state of system between epochs Measurement noise n[k] Process noise v[k] Image Processing and Computer Vision: 8 36

Mathematically How do observations relate to model? Image Processing and Computer Vision: 8 37

Prediction of System State n Relates system states at epochs k and k+1 Image Processing and Computer Vision: 8 38

Prediction of Observation n From predicted system state, estimate what observation should occur: Image Processing and Computer Vision: 8 39

Updating the Model n n Predict/estimate a measurement Make a measurement Predict state of model How does the new measurement contribute to updating the model? Image Processing and Computer Vision: 8 40

Updating the Model n G is Kalman Gain n Derived from A, H, v, n. Image Processing and Computer Vision: 8 41

Example n n n Tracking two corners of a minimum bounding box Matching using colour Image differencing to locate target Image Processing and Computer Vision: 8 42

Condensation Tracking n n So far considered single motions What if movements change? n n n Bouncing ball Human movements Use multiple models n plus a model selection mechanism Image Processing and Computer Vision: 8 43

Selection and Tracking n n Occur simultaneously Maintain n n Predict n n A distribution of likely object positions plus weights Select N samples, predict locations Verify n n Match predicted locations against image Update distributions Image Processing and Computer Vision: 8 44

Tracking Using Hidden Markov Models n Hidden Markov model describes n n States occupied by a system Possible transitions between states Probabilities of transitions How to recognise a transition Image Processing and Computer Vision: 8 45

Optic Flow n n n Assume each pixel moves but does not change intensity Pixel at location (x, y) in image 1 is pixel at (x+Dx, y+Dy) in image 2. Optic flow associates displacement vector with each pixel Image Processing and Computer Vision: 8 46

Image Processing and Computer Vision: 8 47

Aperture Problem I/ x horizontal difference I/ y vertical difference I/ t difference between images One equation, two unknowns cannot solve equation Could solve for movement perpendicular to gradient Image Processing and Computer Vision: 8 48

Solution Impose smoothness constraint: Minimise total of sum of squares of velocity component gradients Image Processing and Computer Vision: 8 49

l is a constant Iterate over a pair of frames, or over a sequence Image Processing and Computer Vision: 8 50

Critique n Assumptions n n n Linear variation of intensities Velocity changes smoothly These are invalid n Especially at object boundaries Image Processing and Computer Vision: 8 51

Area Based Methods n n Match small regions in image 1 with small regions in image 2 Assume objects move but do not deform n Same formulation as for optical flow, different areas involved…. Image Processing and Computer Vision: 8 52

Matching Compute (Dx, Dy) by finding (u, v) that minimises Then (u, v) = (Dx, Dy) Image Processing and Computer Vision: 8 53

Summary n Target acquisition n Image differencing Background model Target following n n Matching Minimum path curvature Model based methods Optic flow Image Processing and Computer Vision: 8 54

This “telephone” has too many shortcomings to be seriously considered as a means of communication. The device is of no value to us. Western Union internal memo, 1876 Image Processing and Computer Vision: 8 55