Introduction to Image Processing and Computer Vision Rahul

Introduction to Image Processing and Computer Vision Rahul Sukthankar Intel Research Laboratory at Pittsburgh and The Robotics Institute, Carnegie Mellon rahuls@cs. cmu. edu

Image Processing vs. Computer Vision • Image processing: s Image image s e. g. , de-noising, compression, edge detection • Computer vision: s Image symbols s e. g. , face recognition, object tracking • Most real-world applications combine techniques from both categories Rahul Sukthankar 15 -829 Lecture 4

Outline • • • Operations on a single image Operations on an image sequence Multiple cameras Extracting semantics from images Applications Rahul Sukthankar 15 -829 Lecture 4

Outline • • • Operations on a single image Operations on an image sequence Multiple cameras Extracting semantics from images Applications Rahul Sukthankar 15 -829 Lecture 4

What is an Image? • 2 D array of pixels • Binary image (bitmap) s Pixels are bits • Grayscale image s Pixels are scalars s Typically 8 bits (0. . 255) • Color images s Pixels are vectors s Order can vary: RGB, BGR s Sometimes includes Alpha Rahul Sukthankar 15 -829 Lecture 4

What is an Image? • 2 D array of pixels • Binary image (bitmap) s Pixels are bits • Grayscale image s Pixels are scalars s Typically 8 bits (0. . 255) • Color images s Pixels are vectors s Order can vary: RGB, BGR s Sometimes includes Alpha Rahul Sukthankar 15 -829 Lecture 4

What is an Image? • 2 D array of pixels • Binary image (bitmap) s Pixels are bits • Grayscale image s Pixels are scalars s Typically 8 bits (0. . 255) • Color images s Pixels are vectors s Order can vary: RGB, BGR s Sometimes includes Alpha Rahul Sukthankar 15 -829 Lecture 4

What is an Image? • 2 D array of pixels • Binary image (bitmap) s Pixels are bits • Grayscale image s Pixels are scalars s Typically 8 bits (0. . 255) • Color images s Pixels are vectors s Order can vary: RGB, BGR s Sometimes includes Alpha Rahul Sukthankar 15 -829 Lecture 4

What is an Image? • 2 D array of pixels • Binary image (bitmap) s Pixels are bits • Grayscale image s Pixels are scalars s Typically 8 bits (0. . 255) • Color images s Pixels are vectors s Order can vary: RGB, BGR s Sometimes includes Alpha Rahul Sukthankar 15 -829 Lecture 4

Canny Edge Detector cv. Canny(…) Images courtesy of Open. CV tutorial at CVPR-2001 Rahul Sukthankar 15 -829 Lecture 4

Morphological Operations • Simple morphological operations on binary images: s erosion: any pixel with 0 neighbor becomes 0 s dilation: any pixel with 1 neighbor becomes 1 • Compound morphological operations: (composed of sequences of simple morphological ops) s s s opening closing morphological gradient top hat black hat • Aside: what is the “right” definition of “neighbor”? Rahul Sukthankar 15 -829 Lecture 4

Morphological Operations Image I Erosion I B Dilatation I B Closing I • B= (I B) B Grad(I)= (I B)-(I B) Top. Hat(I)= I - (I B) Images courtesy of Open. CV tutorial at CVPR-2001 Opening Io. B= (I B) B Black. Hat(I)= (I B)-I Rahul Sukthankar 15 -829 Lecture 4

Hough Transform Goal: Finding straight lines in an edge image Original image Images courtesy of Open. CV tutorial at CVPR-2001 Canny edge + Hough xform cv. Hough. Lines(…) Rahul Sukthankar 15 -829 Lecture 4

Distance Transform • Distance for all non-feature points to closest feature point cv. Dist. Transform(…) Images courtesy of Open. CV tutorial at CVPR-2001 Rahul Sukthankar 15 -829 Lecture 4

Flood Filling cv. Flood. Fill(…) grows from given seed point Images courtesy of Open. CV tutorial at CVPR-2001 Rahul Sukthankar 15 -829 Lecture 4

Image Statistics • Statistics are used to summarize the pixel values in a region, typically before making a decision • Some statistics are computed over a single image: s Mean and standard deviation: cv. Avg(…), cv. Avg. Sdv(…) s Smallest and largest intensities: cv. Min. Max. Loc(…) s Moments: cv. Get. Spatial. Moment(…), cv. Get. Central. Moment(…) • Others are computed over pairs/differences of images: s Distances/norms C, L 1, L 2: cv. Norm(…), cv. Norm. Mask(…) s Others are computed over pairs/differences of images: • Histograms: s Multidimensional histograms: (many functions to create/manipulate) s Earth mover distance – compare histograms: cv. Calc. EMD(…) Rahul Sukthankar 15 -829 Lecture 4

Image Pyramids: Coarse to Fine Processing • Gaussian and Laplacian pyramids • Image segmentation by pyramids Images courtesy of Open. CV tutorial at CVPR-2001 Rahul Sukthankar 15 -829 Lecture 4

Image Pyramids: Coarse to Fine Processing Original image Images courtesy of Open. CV tutorial at CVPR-2001 Gaussian Laplacian Rahul Sukthankar 15 -829 Lecture 4

Pyramid-based Color Segmentation Images courtesy of Open. CV tutorial at CVPR-2001 Rahul Sukthankar 15 -829 Lecture 4

Outline • • • Operations on a single image Operations on an image sequence Multiple cameras Extracting semantics from images Applications Rahul Sukthankar 15 -829 Lecture 4

Background Subtraction • Useful when camera is still and background is static or slowly-changing (e. g. , many surveillance tasks) • Basic idea: subtract current image from reference image. Regions with large differences correspond to changes. • Open. CV supports several variants of image differencing: s Average s Standard deviation s Running average: cv. Running. Avg(…) • Can follow up with connected components (segmentation): s could use “union find” or floodfill: cv. Flood. Fill(…) Rahul Sukthankar 15 -829 Lecture 4

Optical Flow • Goal: recover apparent motion vectors between a pair of images -- usually in a video stream • Several optical flow algorithms are available: s s Block matching technique: cv. Calc. Optical. Flow. BM(…) Horn & Schunck technique: cv. Calc. Optical. Flow. HS(…) Lucas & Kanade technique: cv. Calc. Optical. Flow. LK(…) Pyramidal LK algorithm: cv. Calc. Optical. Flow. Pyr. LK(…) Rahul Sukthankar 15 -829 Lecture 4

Active Contours: Tracking by Energy Minimization • Snake energy: • Internal energy: • External energy: cv. Snake. Image(…) Images courtesy of Open. CV tutorial at CVPR-2001 Rahul Sukthankar 15 -829 Lecture 4

Camera Calibration • • Real cameras exhibit radial & tangential distortion: causes problems for some algorithms. First, calibrate by showing a checkerboard at various orientations: cv. Find. Chess. Board. Corner. Guesses() • Then apply an undistorting warp to each image (don’t use a warped checkerboard!) cv. Undistort(…) • If the calibration is poor, the “undistorted” image may be worse than the original. Images courtesy of Open. CV tutorial at CVPR-2001 Rahul Sukthankar 15 -829 Lecture 4

Outline • • • Operations on a single image Operations on an image sequence Multiple cameras Extracting semantics from images Applications Rahul Sukthankar 15 -829 Lecture 4

Stereo Vision • Extract 3 D geometry from multiple views • Points to consider: s feature- vs area-based s strong/weak calibration s processing constraints • No direct support in Open. CV, but building blocks for stereo are there. Rahul Sukthankar 15 -829 Lecture 4

View Morphing Images courtesy of Open. CV tutorial at CVPR-2001 Rahul Sukthankar 15 -829 Lecture 4

Outline • • • Operations on a single image Operations on an image sequence Multiple cameras Extracting semantics from images Applications Rahul Sukthankar 15 -829 Lecture 4

Face Detection Images courtesy of Mike Jones & Paul Viola Rahul Sukthankar 15 -829 Lecture 4

Classical Face Detection Large Scale Small Scale Images courtesy of Mike Jones & Paul Viola Painful! Rahul Sukthankar 15 -829 Lecture 4

Viola/Jones Face Detector • Technical advantages: s Uses lots of very simple box features, enabling an efficient image representation s Scales features rather than source image s Cascaded classifier is very fast on non-faces • Practical benefits: s Very fast, compact footprint s You don’t have to implement it! (should be in latest version of Open. CV) Rahul Sukthankar 15 -829 Lecture 4

Principal Components Analysis High-dimensional data Lower-dimensional subspace cv. Calc. Eigen. Objects(…) Images courtesy of Open. CV tutorial at CVPR-2001 Rahul Sukthankar 15 -829 Lecture 4

PCA for Object Recognition Images courtesy of Open. CV tutorial at CVPR-2001 Rahul Sukthankar 15 -829 Lecture 4

PCA for Object Recognition Images courtesy of Open. CV tutorial at CVPR-2001 Rahul Sukthankar 15 -829 Lecture 4

Outline • • • Operations on a single image Operations on an image sequence Multiple cameras Extracting semantics from images Applications Rahul Sukthankar 15 -829 Lecture 4

Examples of Simple Vision Systems Shadow Elimination • Idea: remove shadows from projected displays using multiple projectors • Open. CV Techniques: s s Image differencing Image warping Convolution filters Matrix manipulation Poster. Cam • Idea: put cameras in posters and identify who reads which poster • Open. CV Techniques: s Face detection s Face recognition s Unsupervised clustering Rahul Sukthankar 15 -829 Lecture 4

Single Projector: Severe Shadows display screen P Rahul Sukthankar 15 -829 Lecture 4

Two Projectors: Shadows Muted display screen P-1 P-2 Rahul Sukthankar 15 -829 Lecture 4

Dynamic Shadow Elimination display screen camera P-1 P-2 Rahul Sukthankar 15 -829 Lecture 4

Shadow Elimination: Challenges display screen camera P-1 • • P-2 Occlusion detection: what does a shadow look like? Geometric issues: which projectors are occluded? Photometric issues: how much light removes a shadow? Performance: how can we do this in near real-time? Rahul Sukthankar 15 -829 Lecture 4

Shadow Elimination: Solutions display screen camera P-1 • • P-2 Occlusion detection: difference image analysis Geometric issues: single shadow-mask for all projectors! Photometric issues: uncalibrated – feedback system Performance: only modify texture map alpha values Rahul Sukthankar 15 -829 Lecture 4

Shadow Removal with a Single Mask Rahul Sukthankar 15 -829 Lecture 4

Shadow Elimination Algorithm Camera images Projected Rahul Sukthankar 15 -829 Lecture 4

Poster. Cam Overview • Poster. Cam Hardware: s Camera in each poster s Embedded computer in each poster (~ i. PAQ) s Network connection to other posters Rahul Sukthankar 15 -829 Lecture 4

Poster. Cam Details • Face detection: Viola/Jones (no float ops) • Lighting compensation: histogram equalization • Pose variation: additional synthetic faces • Unsupervised clustering: k-means and nearest neighbor with nonstandard distance metric Rahul Sukthankar 15 -829 Lecture 4

Tips on Image Processing and Coding with Open. CV • Use the Open. CV documentation only as a guide (it is inconsistent with the code) • Read cv. h before writing any code • Open. CV matrix functions work on images: e. g. , cv. Sub(…) • Beware camera distortion: cv. Un. Distort(…) may help • Beware illumination changes: s disable auto gain control (AGC) in your camera if you are doing background subtraction s histogram equalization (often good for object recognition) • Image processing algorithms may require parameter tuning: collect data and tweak until you get good results Rahul Sukthankar 15 -829 Lecture 4

Reference Reading • Digital Image Processing Gonzalez & Woods, Addison-Wesley 2002 • Computer Vision Shapiro & Stockman, Prentice-Hall 2001 • Computer Vision: A Modern Approach Forsyth & Ponce, Prentice-Hall 2002 • Introductory Techniques for 3 D Computer Vision Trucco & Verri, Prentice-Hall 1998 Rahul Sukthankar 15 -829 Lecture 4

The End Acknowledgments • Significant portions of this lecture were derived from the Intel Open. CV tutorial by Gary Bradski et al. at CVPR-2001 • Thanks to my former colleagues at Compaq/HP CRL for additional slides and suggestions: Tat-Jen Cham, Mike Jones, Vladimir Pavlovic, Jim Rehg, Gita Sukthankar, Nuno Vasconcelos, Paul Viola Contact rahuls@cs. cmu. edu if you need more information Rahul Sukthankar 15 -829 Lecture 4