Leow Wee Kheng CS 4243 Computer Vision and

Leow Wee Kheng CS 4243 Computer Vision and Pattern Recognition Motion Tracking CS 4243 Motion Tracking 1

Changes are everywhere! CS 4243 Motion Tracking 2

Illumination change CS 4243 Motion Tracking 3

Shape change CS 4243 Motion Tracking 4

Object motion CS 4243 Motion Tracking 5

Camera motion CS 4243 Motion Tracking 6

Object & camera motion CS 4243 Motion Tracking 7

Everything changes! CS 4243 Motion Tracking 8

Motion analysis is tough in general! We focus on object / camera motion. CS 4243 Motion Tracking 9

Change Detection Detects any change in two video frames. Straightforward Compute difference between corresponding pixels: : intensity / colour at (x, y) in frame t. > threshold, has large difference. If CS 4243 method: Motion Tracking 10

Any difference? No CS 4243 Motion Tracking 11

Any difference? Yes, illumination change CS 4243 Motion Tracking 12

Any difference? Yes, position change CS 4243 Motion Tracking 13

Change Detection Can detect Illumination change Position change Illumination and position change But, cannot distinguish between them. Need CS 4243 to detect and measure position change. Motion Tracking 14

Motion Tracking Two approaches Feature-based Intensity CS 4243 gradient-based Motion Tracking 15

Feature-based Motion Tracking Look for distinct features that change positions. Eagle’s wing tips change positions. Tree tops don’t change positions. CS 4243 Motion Tracking 16

Basic Ideas 1. Look for distinct features in current frame. 2. For each feature Search for matching feature within neighbourhood in next frame. Difference in positions displacement. Velocity = displacement / time difference. CS 4243 Motion Tracking 17

Basic Ideas displacement feature area search area CS 4243 Motion Tracking 18

What feature to use? Harris corner Tomasi’s Feature feature descriptors: SIFT, SURF, GLOH, etc. Others CS 4243 Motion Tracking 19

Summary Simple algorithm. Can be slow if search area is large. Can constrain search area with prior knowledge. CS 4243 Motion Tracking 20

Gradient-based Motion Tracking Two basic assumptions Intensity changes smoothly with position. Intensity of object doesn’t change over time. Suppose an object is in motion. Change position (dx, dy) over time dt. Then, from 2 nd assumption: Apply CS 4243 Taylor’s series expansion: Motion Tracking 21

Omit higher order terms and divide by dt Denote Then, u, v are unknown. 2 unknowns, 1 equation: can’t solve! CS 4243 Motion Tracking 22

Any difference in motion? CS 4243 Motion Tracking 23

Any difference in motion? CS 4243 Motion Tracking 24

Any difference in motion? CS 4243 Motion Tracking 25

Any difference in motion? CS 4243 Motion Tracking 26

Any difference in motion? CS 4243 Motion Tracking 27

Any difference in motion? CS 4243 Motion Tracking 28

Aperture Problem Homogeneous region Ix = Iy = It = 0. No change in local region. Cannot detect motion. CS 4243 Motion Tracking 29

Aperture Problem Edge Ix and Iy are zero along edge Cannot measure motion tangential to edge Ix and Iy are non-zero normal to edge Can measure motion normal to edge So, cannot measure actual motion CS 4243 Motion Tracking 30

Aperture Problem Corner Ix and Iy are non-zero in two perpendicular directions 2 unknowns, 2 equations Can measure actual motion CS 4243 Motion Tracking 31

Lucas-Kanade Method Consider two consecutive image frames I and J: I J x+d x x d Object moves from x = (x, y)T to x + d. d = (u, v) T CS 4243 Motion Tracking 32

So, Or Due to noise, there’s an error at position x: Sum error over small window W at position x: Similar to template matching CS 4243 weight Motion Tracking 33

If E is small, patterns in I and J match well. So, find d that minimises E: E / d = 0, compute d that minimises E. First, expand I(x – d) by Taylor’s series expansion: Set Omit higher order terms: Write in matrix form: Intensity gradient CS 4243 Motion Tracking 34

Now, error E at position x is: Now, differentiate E with respect to d (exercise): Setting E / d = 0 gives b the only unknown Z CS 4243 Motion Tracking 35

So, 2 we get unknowns, 2 equations. Can solve for d = (u, v). What happen to aperture problem? Did it disappear? CS 4243 Motion Tracking 36

Lucas-Kanade + Tomasi Lucas-Kanade algorithm is often used with Tomasi’s feature Apply Tomasi’s method to detect good features. Apply LK method to compute d for each pixel. Accept d only for good features. CS 4243 Motion Tracking 37

Example Can you spot tracking errors? CS 4243 Motion Tracking 38

Constraints Math In implementation, W is also small. LK CS 4243 of LK tracker assumes d is small. tracker is good only for small displacement. Motion Tracking 39

How to handle large displacement? What if we scale down images? Displacements CS 4243 are smaller! Motion Tracking 40

Image Pyramid Usually, smoothen image with Gaussian filter before scaling downsample scaled image input image CS 4243 Motion Tracking 41

LK Tracking with Image Pyramid CS 4243 Motion Tracking 42

Construct Apply image pyramids. LK tracker to low-resolution images. Propagate Apply CS 4243 results to higher-resolution images. LK tracker to higher-resolution images. Motion Tracking 43

Example Tracking results are more accurate. CS 4243 Motion Tracking 44

Summary Efficient algorithm, no explicit search. Has aperture problem; track good features only LK tracker can’t track large displacement. Use CS 4243 LK + image pyramid for large displacement. Motion Tracking 45

Software Open. CV [Bir] CS 4243 supports LK and LK with pyramid. offers LK with Tomasi’s features & pyramid. Motion Tracking 46

Appendix Calculation of Ix, Iy, It Use finite difference method Forward difference Ix = I(x 1, y, t) I(x, y, t) Iy = I(x, y 1, t) I(x, y, t) It = I(x, y, t 1) I(x, y, t) Backward difference Ix = I(x, y, t) I(x 1, y, t) Iy = I(x, y, t) I(x, y 1, t) It = I(x, y, t) I(x, y, t 1) CS 4243 Motion Tracking 47

Further Readings Lucas-Kanade tracking with pyramid: [BK 08] Chapter 10. Optical flow: [Sze 10] Section 8. 4. Hierarchical motion estimation (with image pyramid): [Sze 10] Section 8. 1. 1. CS 4243 Motion Tracking 48

References [Bir] S. Birchfield. KLT: An implementation of the Kanade-Lucas. Tomasi feature tracker. http: //vision. stanford. edu/~birch/klt/. [BK 08] Bradski and Kaehler. Learning Open. CV: Computer Vision with the Open. CV Library. O’Reilly, 2008. [LK 81] B. D. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In Proceedings of 7 th International Joint Conference on Artificial Intelligence, pages 674– 679, 1981. [ST 94] J. Shi and C. Tomasi. Good features to track. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 593– 600, 1994. [Sze 10] R. Szeliski. Computer Vision: Algorithms and Applications. Springer, 2010. CS 4243 Motion Tracking 49

References CS 4243 [TK 91] C. Tomasi and T. Kanade. Detection and tracking of point features. Technical Report CMU-CS-91 -132, School of Computer Science, Carnegie Mellon University, 1991. Motion Tracking 50