SuperResolution Authors Wagner Waagen and Cassabaum Presented By

Super-Resolution Authors: Wagner, Waagen and Cassabaum Presented By: Mukul Apte

Introduction n n Definition: Create High Resolution visual output from Low Resolution visual input. Mathematical assistance to features viz. motion detection, face recognition, person detection. Action Packed Sports Images ¨ Astronomy ¨ Medical Imaging ¨ Surveillance ¨ n Range of resources: low (single camera, polaroid lenses) to high (super-resolution chips, androids-ASIMO)

Basic idea Given: A set of degraded (warped, blurred, decimated, noised) images: Required: Fusion of the measurements into a higher resolution image:

Types Static Super-Resolution (SSR) - The creation of a single improved image, from the finite measured sequence of images n Dynamic Super-Resolution (SSR) - Low Quality Movie In - High Quality Movie Out n

Simple Example Concepts I: Raster, CCD, Image in mathematical domain, Matlab n Concepts II: Bresenham’s line algorithm, non-uniform interpolation, frequency domain n

Simple Example For a given band-limited image, the Nyquist sampling theorem states that if a uniform sampling is fine enough (pixel density = twice highest static frequency), perfect reconstruction is possible. D D

Simple Example Due to our limited camera resolution, we sample using an insufficient 2 D grid 2 D 2 D

Simple Example However, we are allowed to take a second picture and so, shifting the camera ‘slightly to the right’ we obtain 2 D 2 D

Simple Example Similarly, by shifting down we get a third image 2 D 2 D

Simple Example And finally, by shifting down and to the right we get the fourth image 2 D 2 D

Simple Example - Conclusion It is trivial to see that interlacing the four images, we get that the desired resolution is obtained, and thus perfect reconstruction is guaranteed. This is Super. Resolution in its simplest form

Uncontrolled Displacements In the previous example we counted on exact movement of the camera by D in each direction. What if the camera displacement is uncontrolled?

Uncontrolled Displacements It turns out that there is a sampling theorem due to Yen (1956) and Papoulis (1977) covering this case, guaranteeing perfect reconstruction for periodic uniform sampling if the sampling density is high enough (1 sample per each D-by-D square).

Uncontrolled Rotation/Scale/Disp. In the previous examples we restricted the camera to move horizontally/vertically parallel to the photograph object. What if the camera rotates? Gets closer to the object (zoom)?

Uncontrolled Rotation/Scale/Disp. There is no sampling theorem covering this case

Further Complications n n n Sampling is not a point operation – there is a blur Motion may include perspective warp, local motion, etc. Samples may be noisy – any reconstruction process must take that into account.

Static Super-Resolution Low Resolution Measurements Static Super-Resolution Algorithm High Resolution Reconstructed Image t

Dynamic Super-Resolution Low Resolution Measurements t Dynamic Super-Resolution Algorithm High Resolution Reconstructed Images t

Approach n n n Image Registration Motion Estimation Projection onto High-Resolution Grid ¨ Non-Uniform Interpolation ¨ Frequency domain – alias correction Projection Registration Low-res Images Registration (sub-pixel grid) High Res Grid

1. 1 Registration (angle) • Rotation Calculation • Correlate 1 st LR image with all LR images at all angles OR • Calculate energy at all angles for all LR images. Correlate energy vector to find the rotation angle Anglei = max index(correlation(I 1(θ), Ii (θ))) LR image 1 LR image 2 i = 2, 3, . . , N (number of LR images) Energy at angle Ii(θ) Energy at angle I 2(θ)

1. 1 Registration (shift) n Shift Calculated using Frequency Domain Method Fi (u. T) = ej 2πuΔs. F 1(u. T) Δs [Δx Δy]T Δs = angle( Fi (u. T) / F 1 (u. T) ) u [fx fy] 2πu • Used only 6% lower u (high freq could be aliased) • Used least square to calculate Δs

2. 1 Frequency Domain n Input: Down-sampled aliased images Goal I: Correct the low-freq aliased data Goal II: Predict the lost high freq values -π π Original High-Res -π π Down-sampled Aliased (fix it) Lost (find it) -π/2 Up-sampled π -π π Desired High-Res

2. 2 Projection onto High-res grid n Papoulis-Gerchberg Algorithm ¨ Projection onto convex sets n Known pixel values n Known Cut-off freq in the HR image ¨ Algorithm: I (known pixel positions) = Known Values I_fft = fft 2(I) I_fft(higher Freq) = 0 I= ifft 2 (I_fft)

SSR – The Model Geometric Warp Blur High. Resolution Image F 1=I H 1 Decimation Y 1 D 1 V 1 X Additive Noise FN HN YN DN VN Low. Resolution Images

Warp – Linear Operation Z X Per every point in X find a matching point in Z 0 F[j, i]=1 0

Model Assumptions We assume that the images Yk and the operators Hk, Dk, Fk, & Wk are known to us, and we use them for the recovery of X. Yk – The measured images (noisy, blurry, down-sampled. . ) Hk – The blur can be extracted from the camera characteristics Dk – The decimation is dictated by the required resolution ratio Fk – The warp can be estimated using motion estimation Wk – The noise covariance can be extracted from the camera characteristics

Special Condition - Noiseless Clearly, this linear system of equations should have more equations than unknowns in order to make it possible to have a unique solution.

SSR – Handling Problems Single image de-noising Single image restoration Single image scaling Motion compensation average Using

SSR Standard Equation Back projection Simulated error Weighted edges All the above operations can be interpreted as iterative operations performed on images. (Typically 15 -20 iterations are performed. )

Thumb Rule on Desired Resolution n Assume that we have N images of M-by-M pixels, and we would like to produce an image X of size L-by-L. Then –

Papoulis – Gerchberg Algorithm Initial Setup FFT (Initial image) Taj Mahal – Low-res image I

Papoulis – Gerchberg Algorithm Known Pixel Values Image at iteration 0 Image after 1 st iteration FFT I(high freq) =0

Papoulis – Gerchberg Algorithm Known Pixel Values Image at iteration 1 Image after 10 iterations FFT I(high freq) =0

Papoulis – Gerchberg Algorithm After 50 iterations Bilinear Interpolation Bicubic Interpolation SR Reconstructed image

Results (Images - I) • Input: 4 snaps using a high-res digital camera • Cropped the same part of each image • SR algorithm compared with bicubic interpolation Original Low-res images (Courtesy: Patrick Vandewalle)

Results (Images - I) Bicubic Interpolation

Results (Images - I) Super-resolution

Results (Images - II) Low-Res Image II • SR Algorithm didn’t work as expected !!! • Reason: • Motion was not restricted to shifts & rotation • Images had affine mapping • Rule I - Need Correct Registration

Results (Frames - I) Original Bicubic SR Why didn’t SR work? ? ? • Low-res images were created by forcing shifts at critical velocities • Rule II - If low-res frames are at critical velocities, can’t create good high-res frame

Results (Frames - II) Original Bicubic SR Why did SR work so well? ? ? • Low-res images were created by forcing shifts at noncritical velocities • Rule III If low-res images have all the info about high-res then HR image can be perfectly constructed

Future Work • Superresolution with multiple motions between frames - create high res video • Predict the high-res frequency components using wavelet methods Predict

Acknowledgements Prof John Apostolopoulos n Prof Susie Wee n Patrick Vandewalle n

THANK YOU!!! QUESTIONS? COMMENTS?