RealTime Tracking Of Objects In UAV Video Imagery

Real-Time Tracking Of Objects In UAV Video Imagery Dr. Shuqun Zhang Binghamton University IFEC August 19 2003

Object Tracking Overview • • • A lot amount of research has been performed. It remains a very challenge task. Difficulties come from – – • No prior knowledge about video and object Object may change dramatically during moving Most successful tracking systems claimed require – – – stationary camera slowly moving but not stationary object a large number of pixels on object simple background … 2

UAV testing videos • • Video types: optical and IR Resolution: 704 x 480 or 320 x 240 FPS: 30 or 15 frames/second Camera may move very fast – • Object may also move fast – • More than 10 pixels/frame Object size – • More than 14 pixels/frame varies from 5 x 5 to 90 x 90 Background – cluttered 3

Existing object tracking methods (1) • Recognition-based tracking – – Match object template/model – Suitable for UAV objects? Example: correlation-based recognition • • No, if tracking of small targets is needed. Computationally expensive 4

Existing object tracking methods (2) • Feature-based tracking – track image features such as corners, lines, – Problems • • – Need sufficient number of “good” features How to determine which features belong to the same object? Suitable for UAV objects? • Not for small targets 5

Existing object tracking methods (3) • Optical flow-based tracking – Moving object has different velocity from the background – Problems • • • – Expensive operation Sensitive to noise Difficult to obtain accurate optical flow Suitable for UAV objects? • No 6

Existing object tracking methods (4) • Contour-based tracking – Segment object and track object contour – Problems • • – Need to provide initial object boundary manually Computationally expensive Suitable for UAV objects? • no 7

Existing object tracking methods (5) • Region-based tracking – – Segment object by temporal differencing – Problem track object regions • – Can be distracted easily by noise or erroneous motion estimates Suitable for UAV objects? • yes 8

About this project • Develop an algorithm to track an object mouse clicked by a user in real-time. Can you click on these objects every time? • Main differences from other object tracking projects – – – Track both moving and stationary objects Track both tiny and large objects Used for video annotation purpose 9

Block diagram Frame n+1 Frame n Make static object “move” Motion Detection Object tracking Image registration Moving object segmentation Motion prediction 10

Image registration • Camera motion estimation – • Camera may move fast ( >14 pixels/frame) – • a 3 -parameter model (translation + rotation) is used Use iterative motion estimation To reduce computation time – Image is downsampled before motion estimation 11

Motion detection for moving objects • • Temporal change detection – differencing – thresholding How to determine threshold T? – – empirically or automatically: T = 2. 5 3 12

Change detection • Extracts both object region and noise • • Incomplete object region Noise – – • Camera noise Motion noise (due to inaccurate motion estimation or insufficient modeling) Difficulty – how to separate object regions from noise 13

More change detection results • Performance depends on image quality, background, and object size 14

Observations • • Small objects – – More noise – Object pixels are connected or close to each other Noise distributed close to the object, even connected to the object Large objects – – – Less noise Noise distributed far from the object Object broken into several regions and they might be far away from each other. 15

Identifying object region • Object region is the region closet to the center of the image (mouse click point). • It works very well with small objects 16

When object is large, • Only a small piece of the object is extracted. Image difference Size filtering • thresholding Object region Further spatial segmentation is required. 17

Edge-based segmentation for large objects • Sobel operator is used in edge detection • Note: – – the segmentation is far from perfect. But it is general sufficient for tracking purpose since we need the approximate centroid and bounding box of the object only. 18

How to track stationary objects? • • Change detection can not detect stationary objects. Possible solutions: – Use another algorithm for stationary objects • – • Problem: hard to separate stationary object from small object Make stationary object “move” How to make stationary objects “move”? – By shrinking object (according to change detection principle) – A 3 x 3 mean filter is used in this project 19

Data flow of the proposed method Play video 720 x 480 Mouse click on object Crop current and next frames Very likely click on background Two 101 x 101 images assume biggest object size < 90 x 90 20

Data flow (continued) Denoising & moving static object Current frame Average filtering next frame Registration subtraction > threshold 2. 5 21

Data flow (continued) Size filtering Morphological closing no yes Area <T Mouse click point in any region? yes Centroid calculation no Find region(s) closest to that point 22

Data flow (continued) Edge detection Select corresponding edge Thicken “AND” with Centroid & Bounding box Motion prediction Region closest to the mouse point 23

Parameters to determine 1. Threshold for change subtraction (2. 7 ) 1. 2. Threshold for size filtering (6) 1. 3. Use the largest expected motion Threshold for separating small and large objects (80) 1. 5. Depends on the size of the smallest object Search range for the closest region to the mouse clicked point (10) 1. 4. 2. 5 - 3. Depends on the size of the smallest object expected Others: structuring elements for morphological filtering operations (3 x 3) 24

Object tracking • • Predict the new location of object Methods: – Kalman filter – Curve fitting and prediction – Constant velocity 25

Advantages and restrictions of the algorithm • Advantages: – Low computational complexity • – • image registration (? %), video decoding (? %), change detection (? %), edge-based segmentation (? ) Tracks various types and sizes of objects. Restrictions – – – Object size < window size (101 x 101) Objects cannot be too close to each other No occlusion 26

Improving algorithm robustness • Information used – – – • Spatial: edge detection Temporal: image differencing Human: mouse click Information could be used – – – History of motion and object regions Better region representation methods Color 27

Summary • Develop a real-time object tracking algorithm for video annotation purpose. • The approach is based on spatio-temporal filtering. • • The algorithm in implemented in Matlab. • Further research work is needed to improve the robustness of the algorithm. It tracks well when the mouse click point is close enough to object. 28

Demo 29