Northeastern University Fall 2005 CSG 242 Computational Photography

Northeastern University, Fall 2005 CSG 242: Computational Photography Ramesh Raskar Mitsubishi Electric Research Labs Northeastern University Nov 30, 2005 Course Web. Page : http: //www. merl. com/people/raskar/photo/course/ Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Plan for Today • • • Assignment 4 Paper reading Topics – Tools: Gradient Domain and Graph Cuts – Type of Sensors • • • Project Updates Peter Sand, Video Special Effects Course feedback Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Reading Paper Presentations 15 minutes: 10 minute presentation, 5 minutes for discussion Use Powerpoint slides, Bring your own laptop or put the slides on a USB drive, (print the slides to be safe) Format: Motivation Approach (New Contributions) Results Your own view of what is useful, what are the limitations Your ideas on improvements to the technique or new applications (atleast 2 new ideas) It is difficult to explain all the technical details in 15 minutes. So focus on the key concepts and give an intuition about what is new here. Ignore second order details in the paper, instead describe them in the context of the results. Keep the description of the approach simple, a rule of thumb: no more than 3 equations in your presentation. Most authors below have the powerpoint slides on their websites, so feel free to use those slides and modify them. Be careful, do not simply present all their slides in sequence. You should focus on only the key concepts and add your own views. If the slides are not available on the author website, copy paste images from the PDF to create your slides. Sometimes you can send email to the author, and s/he will send you the slides. Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

A Introduction Digital photography compared to film photography Image formation, Image sensors and Optics B Understanding the Camera Parameters: Pixel Resolution, Exposure, Aperture, Focus, Color depth, Dynamic range Nonlinearities: Color response, Bayer pattern, White balance, Frequency response Noise: Electronic sources Time factor: Lag, Motion blur, Iris Flash settings and operation Filters: Polarization, Density, Decamired In camera techniques: Auto gain and white balance, Auto focus techniques, Bracketing C Image Processing and Reconstruction Tools Convolution, Overview Gradient domain operations, Applications in fusion, tone mapping and matting Graph cuts, Applications in segmentation and mosaicing Bilateral and Trilateral filters, Applications in image enhancement Today D Improving Performance of Camera Dynamic range: Variable exposure imaging and tone mapping, Frame rate: High speed imaging using multiple cameras Pixel resolution: Super-resolution using jitter Focus: Synthetic Aperture from camera array for controlled depth of field E Image Processing and Reconstruction Techniques Brief overview of Computer Vision techniques: Photometric stereo, Depth from defocus, Defogging Scene understanding: Depth edges using multiple flashes, Reflectance using retinex Denoising using flash and no flash image pairs Multi-image fusion techniques: Fusing images taken by varying focus, exposure, view, wavelength, polarization or illumination Photomontage of time lapse images Matting Omnidirectional and panoramic imaging F Computational Imaging beyond Photography Optical tomography, Imaging beyond visible spectrum, Coded aperture imaging, multiplex imaging, Wavefront coded Microscopy Scientific imaging in astronomy, medicine and geophysics Next week G Future of Smart and Unconventional Cameras Overview of HDR cameras: Spatially adaptive prototypes, Log, Pixim, Smal Foveon X 3 color imaging Programmable SIMD camera, Jenoptik, IVP Ranger Gradient sensing camera Demodulating cameras (Sony IDcam, Phoci) Future directions Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Tentative Schedule • Nov 30 th – Project Update – Special lecture: Video Special Effects • Dec 2 nd (Friday) – Hw 4 due Midnite • Dec 7 th – Computational Imaging Beyond Photography – Special lecture: Mok 3 • Dec 15 th (Exam week) – In class exam (instead of Hw 5) – Final Project Presentation Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Assignment 4: Playing with Epsilon Views See course webpage for details • Resynthesizing images from epsilon views (rebinning of rays) • http: //groups. csail. mit. edu/graphics/pubs/siggraph 2000_drlf. pdf In this assignment, you will use multiple pictures under slightly varying position to create a large synthetic aperture and multiple-center-of-projection (MCOP) images You will create (i) An image with programmable plane of focus (ii) A see-through effect ----------------------------(A) Available set http: //www. eecis. udel. edu/~yu/Teaching/toy. LF. zip Use only 16 images along the horizontal translation (B) Your own data set Take atleast 12 -16 pictures by translating a camera (push broom) The forground scene is a flat striped paper Background scene is a flat book cover or painting Choose objects with vibrant bright saturated colors Instead of translating the camera, you may find it easier to translate the scene Put the digital camera in remote capture time lapse interval mode (5 second interval) Effect 1: Programmable focus Rebin rays to focus on first plane Rebin rays to focus on back plane but rejecting first plane Effect 2: MCOP images Rebin rays to create a single image with multiple views Useful links http: //groups. csail. mit. edu/graphics/pubs/siggraph 2000_drlf. pdf Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Image Tools • Gradient domain operations, – Applications in tone mapping, fusion and matting • Graph cuts, – Applications in segmentation and mosaicing • Bilateral and Trilateral filters, – Applications in image enhancement Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

But, Simple Pixel Blending Creates Ugly Artifacts Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Pixel Blending Our Method: Integration of blended Gradients Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Gradient field Nighttime image x I 1 G 1 Y G 1 Mixed gradient field Y x G G Importance image W I 2 Y Final result x G 2 Daytime image Gradient field Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Reconstruction from Gradient Field • Problem: minimize error • Estimate I’ so that |Ñ I’ – G| G = Ñ I’ • Poisson equation GX Ñ 2 I’ = div G • Full multigrid solver GY I’ Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Intensity Gradient in 1 D 105 Intensity 105 1 Gradient 1 I(x) Gradient at x, G(x) = G(x) I(x+1)- I(x) Forward Difference Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Reconstruction from Gradients 105 Gradient 105 ? 1 Intensity ? 1 G(x) I(x) For n intensity values, about n gradients Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Reconstruction from Gradients 105 Gradient 105 Intensity ? 1 1 G(x) I(x) 1 D Integration I(x) = I(x-1) + G(x) Cumulative sum Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Intensity Gradient in 2 D Gradient at x, y as Forward Differences Gx(x, y) = I(x+1 , y)- I(x, y) Gy(x, y) = I(x , y+1)- I(x, y) G(x, y) = (Gx , Gy) Grad X Grad Y Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Intensity Gradient Vectors in Images Gradient Vector Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Reconstruction from Gradients Given G(x, y) How to compute = (Gx , Gy) I(x, y) for the image ? For n 2 image pixels, 2 n 2 gradients ! Grad X 2 D Integration Grad Y Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Intensity Gradient in 2 D Recovering Original Image Grad X 2 D Integration Grad Y Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Intensity Gradient Manipulation Recovering Manipulated Image Grad X Grad Y Gradient Processing New Grad X New Grad Y Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Intensity Gradient Manipulation Recovering Manipulated Image Gradient Processing New Grad X 2 D Integration New Grad Y Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Intensity Gradient Manipulation Recovering Manipulated Image Gradient Processing New Grad X 2 D Integration New Grad Y Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Intensity Gradient Manipulation A Common Pipeline Grad X Grad Y Gradient Processing New Grad X 2 D Integration New Grad Y Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Reconstruction from Gradients Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Euler-Lagrange Equation Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Application: Compressing Dynamic Range How could you put all this information into one Image ? Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Attenuate High Gradients Intensity 105 1 I(x) Gradient 105 1 I(x) G(x) Maintain local detail at the cost of global range Fattal et al Siggraph 2002 Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Basic Assumptions • The eye responds more to local intensity differences (ratios) than global illumination • A HDR image must have some large magnitude gradients • Fine details consist only of smaller magnitude gradients Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Gradient Compression in 1 D Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Gradient Domain Method Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Basic Method • • • Take the log of the luminances Calculate the gradient at each point Scale the magnitudes of the gradients with a progressive scaling function (Large magnitudes are scaled down more than small magnitudes) • Re-integrate the gradients and invert the log to get the final image Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Summary: Intensity Gradient Manipulation Grad X Grad Y Gradient Processing New Grad X 2 D Integration New Grad Y Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Graph and Images Credits: Jianbo Shi Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Brush strokes Computed labeling Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Image objective 0 for any label 0 if red ∞ otherwise Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Graph Based Image Segmentation = Graph partition i Wij j Wij V: graph node Image pixel E: edges connection nodes Link to neighboring pixels Wij: Edge weight Pixel similarity Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Minimum Cost Cuts in a graph Cut: Set of edges whose removal makes a graph disconnected Si, j : Similarity between pixel i and pixel j Cost of a cut, A A Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Graph Cuts for Segmentation and Mosaicing Cut ~ String (loop) on a height field Places where the string rests Brush strokes Computed labeling Ramesh Raskar, Comp. Photo Class Northeastern, Fall 2005

Cameras we consider 2 types : 1. CCD 2. CMOS Department of Computer and Information Science

CCD separate photo sensor at regular positions no scanning charge-coupled devices (CCDs) area CCDs and linear CCDs 2 area architectures : interline transfer and frame transfer photosensitive storage Department of Computer and Information Science

The CCD camera Department of Computer and Information Science

CMOS Same sensor elements as CCD Each photo sensor has its own amplifier More noise (reduced by subtracting ‘black’ image) Lower sensitivity (lower fill rate) Uses standard CMOS technology Allows to put other components on chip ‘Smart’ pixels Foveon 4 k x 4 k sensor 0. 18 process 70 M transistors Department of Computer and Information Science

CCD vs. CMOS • • Mature technology • Specific technology • High production cost • High power consumption • Higher fill rate • Blooming • Sequential readout • • Recent technology Standard IC technology Cheap Low power Less sensitive Per pixel amplification Random pixel access Smart pixels On chip integration with other components Department of Computer and Information Science

Color cameras We consider 3 concepts: 1. Prism (with 3 sensors) 2. Filter mosaic 3. Filter wheel … and X 3 Department of Computer and Information Science

Prism color camera Separate light in 3 beams using dichroic prism Requires 3 sensors & precise alignment Good color separation Department of Computer and Information Science

Prism color camera Department of Computer and Information Science

Filter mosaic Coat filter directly on sensor Demosaicing (obtain full color & full resolution image) Department of Computer and Information Science

Filter wheel Rotate multiple filters in front of lens Allows more than 3 color bands Only suitable for static scenes Department of Computer and Information Science

Prism vs. mosaic vs. wheel approach # sensors Separation Cost Framerate Artefacts Bands Prism 3 High Low 3 High-end cameras Mosaic 1 Average Low High Aliasing 3 Low-end cameras Wheel 1 Good Average Low Motion 3 or more Scientific applications Department of Computer and Information Science

New color CMOS sensor Foveon’s X 3 better image quality smarter pixels Department of Computer and Information Science

Foveon: Thick Sensor

Pixim

High Dynamic Range http: //www. cybergrain. com/tech/hdr/ Fuji's Super. CCD S 3 Pro camera has a chip with high and low sensitivity sensors per pixel location to increase dynamic range

Gradient Camera Sensing Pixel Intensity Difference with Locally Adaptive Gain Ramesh Raskar, MERL Work with Jack Tumblin, Northwestern U, Amit Agrawal, U of Maryland

Natural Scene Properties Intensity Gradient 105 1 1 x x Intensity Histogram 1 5 Gradient Histogram 5 5

Original Image Intensity Camera Image Intensity values ranging from 0 to 1800 Intensity ramp plus low contrast logo 8 bit camera for 1: 1000 range Problem: . saturation at high intensity regions Log Camera Image 8 bit log for 1: 106 range Problem: Visible quantization effects at high intensities Locally Adaptive Gain Pixel divided by the average of local neighborhood. Thus the low frequency contents are lost and only detail remains. Gradient Camera Image In proposed method, we sense intensity differences. We use a 8 bit A/D with resolution of log(1. 02) to capture 2% contrast change between adjacent pixels. Notice that the details at both high and low intensities are captured.

Gradient Camera • Two main features 1. Sense difference between neighboring pixel intensity At each pixel, measure ( x , y ) , x = Ix+1, y - Ix, y , y = Ix, y+1 - Ix, y 2. With locally adaptive gain • Gradient camera is very similar to locally adaptive gain camera • Locally Adaptive Gain Camera – Gain is different for each pixel – Problem: Loses low frequency detail and preserves only high frequency features (edges) • Gradient Camera – The gain is same for four adjacent pixels – Difference between two pixels is measured with same gain on both pixels – Reconstruct original image in software from pixel differences by solving a linear system (solving Poisson Equation)

Camera Pipeline On-board Hardware Difference between pixels Local gain adaptive to difference Software 2 D Integration to reconstruct the image

Detail Preserving Intensity Camera Log Intensity Camera Gradient Camera Intensity cameras capture detail but lose range Log cameras capture range but lose detail

Quantization Intensity Histogram 1 105 Gradient Histogram Original Image Uniform quantization 3 bits -105 Grad. Cam requires fewer bits Log Uniform quantization 3 bits Log Uniform gradients quantization 3 bits In the reconstructed image, error is pushed to high gradient pixel positions which is visually imperceptible

High Dynamic Range Images Scene Intensity camera saturation map Gradient camera saturation map Intensity camera fail to capture range Gradients saturate at very few isolated pixels

3 D Cameras • Time of flight or phase – ZCam – Canesta • Phase Decoding of modulated illumination – Phase difference = depth – Magnitude = reflectance • Structured Light – Binary coded light and triangulation

Novel Sensors • • • Gradient sensing HDR Camera, Log sensing Line-scan Camera Demodulating Motion Capture 3 D

Fantasy Configurations • ‘Cloth-cam’: ‘Wallpaper-cam’ elements 4 D light emission and 4 D capture in the surface of a cloth… • Floating Cam: ad-hoc wireless networks form camera arrays in environment… • Other ray sets: Multilinear cameras, canonical ‘basis’ cameras (linear combination of 8 types) Mc. Millan’ 04, ‘ 05