Computer Vision Class project proposal Brian Clipp COMP

Computer Vision Class project proposal Brian Clipp COMP 256

Computer Vision Kalman Filter Implementation of Bundle Adjustment Brian Clipp • Motivation – Generate most acurate possible estimate of 3 D points and camera poses given multiple views of a scene. • Objective 2

Computer Vision Kalman Filter Implementation of Bundle Adjustment Brian Clipp 3

Computer Vision Kalman Filter Implementation of Bundle Adjustment Brian Clipp • Advantages – Variable bundle adjustment window size – Each new P matrix or point added to the bundle adjustment window does not force re-computation of the entire bundle. – Future P matrices are influenced by P matrices no longer in the window, forcing continuity in the model. 4

Computer Vision Plane Histograms for Planesweeping Stereo David Gallup

Computer Vision Planesweeping Stereo • • • near 6 Hypothesize a plane Project images from all cameras onto it Measure photoconsistency (cost) as seen from a reference view Repeat for a family of planes Per pixel, record plane with best cost far Multiway Planesweep • Sweep planes in multiple directions • Better alignment with surface produces better results • Surface normals known through heuristics or other information

Computer Vision Cost Function and Plane Histogram Cost function • Each pixel has a cost function • Chosen depth is plane of minimum cost • Poorly defined minimum in some cases • Multiple local minima in some cases 7 Histogram • Histogram 3 D points against planes in each direction • Helps to disambiguate poorly defined minima • Provides temporal coherency

Computer Vision 8 Cost Function and Plane Histogram

Computer Vision Occlusion Inference with Visual Hull Li Guan

Computer Vision Probabilistic Occupancy Grid • Jean-Sebastien Franco, “Fusion of Multi-View Silhouette Cues Using a Space Occupancy Grid”, ICCV, 2005 • See the Video 10

Computer Vision 11

Computer Vision Project Goal • To formalize the inference of POccluder from POccupancy Grid For every view: (1 -POccluder)* Pd=POccupancy Grid • To infer occlusion masks for a single view • Similar to the occupancy grid, to have an occlusion grid for the environment 12

Computer Vision Continuous Calibration Tyler Johnson COMP 256 Spring ‘ 06

Computer Vision Calibration in Tiled Displays • Multiple overlapping projectors must be calibrated • Large up-front calibration process • Stereo cameras triangulate display surface and calibrate projectors using correspondences 14

Computer Vision Continuous Calibration • Would like to be able to continuously refine estimate display surface and projector parameters • Structured light vs. feature detection in continuous image streams • Current structured light methods sacrifice image quality -> feature detection/correlation • Requires dynamic scene with available features 15

Computer Vision Proposal • Initial calibration of projector, stereo camera pair and estimate of surface geometry • Detect features in the user imagery observed by camera and track with optical flow • Given calibration, find features in other views • Use matched features to re-estimate disp. surface/projector calibration • Kalman filter aids in estimation and provides level of uncertainty – Predictor-Corrector 16

Computer Vision 3 D Hole Filling Using Texture Synthesis Concepts COMP 256 John Mason

Computer Vision John Mason 3 D Hole Filling Using Texture Synthesis Concepts • 3 D geometry – 3 D by hand • High quality • Takes time and people – 3 D from scans • Tends to have holes where scanner can’t reach or due to other surface qualities • Humans easily distracted by unexpected holes • Multiple scans can’t always fill the holes • How to fill the holes? 18 Images: Davis & Levoy

Computer Vision John Mason 3 D Hole Filling Using Texture Synthesis Concepts • Texture Synthesis – Take a small piece of texture – Make a bigger piece of texture – Fill in gaps in texture • 3 D Hole Filling – Look at hole’s neighborhood – Determine local geometric features – Create a reasonable representation of the missing geometry – Looks like it belongs or exact match? 19 Image: Wei & Levoy Images: Efros & Leung Image: Wei & Levoy Images: Davis & Levoy

Computer Vision John Mason 3 D Hole Filling Using Texture Synthesis Concepts • Project Target – Explore value of various texture synthesis methods applied to 3 D hole filling – Develop rapid, automated hole filling implementation – Make sure filled hole looks like it should belong there – As time allows, compare to other hole filling technologies 20 Prior work in the area of Texture Synthesis, and related disciplines, includes works by Efros & Leung, Efros & Freeman, Turk & Levoy, Wei & Levoy, Kwatra, and Lefebvre & Hoppe among others. There is additional work recreating 3 D textures and surfaces including works by Dong & Chantler, Torrez & Dudek, Ricken & Warren, Velho et al, and Davis & Levoy among others.

Computer Vision Paul Merrell Project Proposal COMP 256

Computer Vision Hole Removal • 3 D Reconstructions from stereo often have holes. • The holes are a result of occlusions. • Goal of my project is to fill in the holes. • Doesn’t need to be perfect, just better than the holes. 22

Computer Vision Scene with an Occlusion 23 Use Smoothness to fill in hole w/o texture Texture Synthesis to Create Texture

Computer Vision 2 D Hole Filling Using Texture Synthesis 3 D Hole Filling 24 Copy from Similar Parts

Computer Vision Parametric (Global) Optic Flow for Stabilization Brad Moore

Computer Vision Goal • Evaluate and implement parametric (global) optical flow – Speed of algorithm – Points of failure 26

Computer Vision Goal (cont. ) • Apply to the problem of stabilization (removing jitters) • Implement a real-time demo 27

Computer Vision Approach • Development on Windows PC • Program in Matlab • Input Device: webcam (purchase, acquire) or video from digital camera • Possible Issues – Speed – Noise (quality of camera) 28

Computer Vision Ultrasound Calibration Tabitha Peck

Computer Vision Ultrasound Calibration Tabitha Peck • Problem: How do you determine the relationship between the location of an ultrasound probe and the real world location of the objects in the image? 30

Computer Vision The Setup Tabitha Peck Ultrasound Probe Optotrak Certus Tracker Image Phantom 31

Computer Vision Challenges Tabitha Peck • Distinguishing items within the phantom • You do not know the exact location of items in the phantom • Error produced by Optotrak Certus 0. 1 mm for x, y coordinates 0. 15 mm for z coordinate • Ultrasound images are not 2 -D 32

Computer Vision Image Segmentation for Display Surface Reconstruction • Overview (WAV): • The goal - Robust automatic calibration of projectors displaying on complex surfaces, using commodity hardware (i. e. cameras, projectors, & PCs) 33 R. Skarbez -- 20 Feb 2006

Computer Vision Image Segmentation for Display Surface Reconstruction • One aspect of this problem is the accurate modeling of the surface geometry • WAV has used several methods: – Tessellation of the 3 D point cloud • Benefits: – Extremely general (Makes no assumptions about the underlying display geometry) • Drawbacks: – Very inefficient (1000 s-100000 s of polys) – Noisy (Small errors make surface look “wavy”) – Does not reconstruct corners well (BIG problem) 34 R. Skarbez -- 20 Feb 2006

Computer Vision Image Segmentation for Display Surface Reconstruction • Previous attempts, cont’d: – Fitting planes to data (Paper in EDT ’ 06) • Benefits: – Many fewer polygons (1 s-10 s) – Much less noise (Surface forced to be planar) – More accurate corner detection than previously • Drawbacks: – Recursively fit on entire point cloud – can generate spurious or redundant planes – Corners still not perfect – dependent on quality of fitted planes – Only works on piecewise-planar display surfaces • Want to get best of all worlds – Generality, efficiency, low noise, and better corner detection • How? Segment the image, and process each segment independently 35 R. Skarbez -- 20 Feb 2006

Computer Vision Image Segmentation for Display Surface Reconstruction • Method – Scan lines across the display surface (using the projectors) – Segment the camera images based on discontinuities in the observed lines – Process each segment individually (e. g. fit a plane if possible, if that fails, try fitting a quadric, or tessellate the data) – Reduces to plane-fitting in the simple case (albeit without spurious planes, and possibly with more accurate corners, due to the fact that they are directly located) – In the general case, allows for handling of arbitrary display surfaces at the cost of generating a more complex display model 36 R. Skarbez -- 20 Feb 2006

Computer Vision Avatars in VR • Jeremy Wendt • Avatars representation of the user in the environment • Improve Presence • Requires tracking several joint locations 37

Computer Vision Camera Tracking • Needs to be fast – 30 fps – latency below 100 ms • Needs to be robust – Jerky motion or unpredictable update break presence • Large tracked area, low cost would be benefits… not necessary 38

Computer Vision Proposed System • Use brightly colored markers • Required (Real Time+UI): – 0. Calibrate cameras – 1. Initialize color markers for cameras – 2. Track the color blob center images – 3. Find 3 space coordinate for tracked position • Bonus: – 4. Smoothing results between frames – 5. Placing avatar at specified location – 6. Use in HMD tracked space 39

Computer Vision Aerial Image registration. Changchang Wu • Image sources – satellites (USGS) – helicopters – GIS (Google Earth, …) – 3 d model screenshot • The matching problem – Matching of different image sets – Matching the images with 3 d model • Images with elevation information – Retrieve the geographical location 40

Computer Vision Image Matching • Local matching of SIFT features – Scale Invariant Feature Transform (Lowe 99) – Select the best match for each feature – Look for local constraints for generating reliable local matches in this particular problem – Look for strategies of selecting more discriminative SIFT features • Global matching – Improve the efficiency and robustness of the current matching algorithm – Implement the matching of images with elevation data, use RANSAC to fit the model. 41

Computer Vision A matching example • 6032 SIFT in the left, and 5591 in the right ( part of them are displayed) 667 putative matches 34 correct matches 42

Computer Vision Normal Recovery from Reflective Surfaces Talha Zaman COMP 256

Computer Vision The Idea • Urban environments often have buildings with large reflective surfaces (plate glass windows) • The reflections from the surfaces contain a large amount of information about the surface geometry • The goal is to use this information to recover the normals and reconstruct the shape of the reflective surface 44

Computer Vision The Approach • Take a picture (or several pictures) of the reflective surface • Take pictures of the reflected scene directly (from same camera position? ) • Figure out the relationships between the surface, the camera, and the scene • Relate reflected image to real image and recover normals • Reconstruct the surface 45