Super Resolution Image Reconstruction Using Kriging Muharrem Mercimek

Super Resolution Image Reconstruction Using Kriging Muharrem Mercimek Lab Presentation 26 February 2009 1

Contents • Super-resolution (SR)? • Components of traditional Super-resolution (SR) image reconstruction algorithms – Sub-pixel Registration Algorithms – Image Reconstruction Algorithms • Kriging • Conclusions , Future Work. 2

Why Super-resolution? • Imaging plays a key role in many diverse application areas. • Imaging devices have limited achievable resolution due to many theoretical restrictions. An original scene with a continous intensity function warps at the camera lens. • Additionally, due to the imperfections of measuring devices instability of the observed scene (motion, media turbulence) acquired images are blurred noisy and corrupted with insufficient spatial temporal resolution – (the resolution w. r. t time. There is a trade off between spatial resolution and temporal resolution) 3

Why Super-resolution? Objective: • In order to recover the original image techniques blind deconvolution and super-resolution are used to remove the blur and to increase the resolution respectively. • For one single observation the problem is ill-posed. Assume we have multiple LR observations, differences between images are very important to provide information abut the original scene. 4

BSR Method in Practical Use Raw Image Sequence Rough Image Registration SR Image Reconstruction Raw Image (Cropped) 2 x Interpolation 2 x Optical zoom (ground truth) 2 x SR image (output) F. Sroubek, F. Cristobal, and J. Flusser, “A unified approach to super-resolution and multichannel blind deconvolution, ” IEEE Trans. on Image Processing, Vol. 16, No. 9, 2007. 5

Contents • Super-resolution (SR)? • Components of traditional Super-resolution (SR) image reconstruction algorithms – Sub-pixel Registration Algorithms – Image Reconstruction Algorithms • Kriging • Conclusions , Future Work. 6

Traditional SR methods 1 - Traditional SR image reconstruction, two stages Motion estimation ; sub-pixel registration Wise interpolation techniques; overlaying the LR images on an HR grid, and interpolating the missing values. 2 -Estimation of shift and rotation parameters. No assumption on PSF during reconstruction. Sub-pixel misalignments during image acquisition 3 -interpolation on a high resolution grid. Instead of usingle image (Interpolation) … Using multiple frames can provide more information of the scene. Differences between the images make the system more stable. Different Image Pixels on the LR grid 7

Motion Estimation algorithms • Super-Resolution (SR) is a process by which a number of LR images are combined into a single HR image, which has a greater resolving power. SR is not only useful to enhance the resolving power of an image; it can also, to some extent, reduce the aliasing noticeably. • Merely upsampling and interpolating the source LR image does not have a high resolution than the source image. Motion Estimation algorithms 1 Vandewalle et al. (EPFL) • This method, developed at the EPFL, • Frequency domain efforts, • Rotation in the space domain is easily visible in the amplitude of the Fourrier Transform. • It has a specific strategy for aliasing seen on LR images. P. Vandewalle, S. Süsstrunk and M. Vetterli, A Frequency Domain Approach to Registration of Aliased Images with Application to Super. Resolution, EURASIP Journal on Applied Signal Processing (special issue on Super-resolution), Vol. 2006, pp. , 14 pages, 2006. 8

Frequency domain motion estimation ( is the half size of the image) 9 P. Vandewalle, S. Süsstrunk and M. Vetterli, A Frequency Domain Approach to Registration of Aliased Images with Application to Super. Resolution, EURASIP Journal on Applied Signal Processing (special issue on Super-resolution), Vol. 2006, pp. , 14 pages, 2006.

Motion Estimation algorithms 2 -Keren et al. • The motion estimation algorithm by Keren et al. is an iterative planar motion estimation algorithm based on Taylor expansion. A pyramidal scheme was is used to increase the precision for the large motion parameters. • Very accurate method for sub-pixel registration. The following image registration has found to be the most accurate and robust by Keren et. al. The reason for not including Fourier transform is reported as it proved to be very sensitive to our noisy environment. f (. ) reference image, g(. ) moving image The image function will be mapped as; using the Taylor series expansion of sine and cosine functions D. Keren, S. Peleg, and R. Brada, “Image sequence enhancement using sub-pixel displacements, ” Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’ 88), pp. 742 -746, 1988. 10

Reconstruction algorithms • When the low-resolution images are accurately registered the samples of the different images can be combined to reconstruct a high resolution image. Reconstruction algorithms 1 Interpolation • This method simply locates all the images' pixels on a High Resolution grid, 2 Iterated Back Projection • The idea behind Iterated Back Projection is to start with a rough estimation of the HR image, and iteratively add to it a "gradient" image, • The sum of the errors between each LR image and the estimated HR image that went through the appropriate transforms is used D. Keren, S. Peleg, and R. Brada, “Image sequence enhancement using sub-pixel displacements, ” Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’ 88), pp. 742 -746, 1988. 11

Reconstruction algorithms 3 Robust Super Resolution (zomet et. al) • • Robust Super Resolution is a more robust version of the above Iterated Back Projection. The only difference resides in the computation of the gradient, which is not given by the sum of all errors, but by the median of all errors. This brings robustness against outliners in the LR images. • 4 - POCS (Patti et. al) • Projection onto Convex Sets (POCS) algorithm defines convex sets representing constraints on the reconstructed image. Estimated reconstruction is successively projected onto different convex sets. • 5 -Kriging A. J. Patti, M. I. Sezan, and A. M. Tekalp, “Super-resolution video reconstruction with arbitrary sampling lattices and nonzero aperture time, ” IEEE Trans. on Image Processing, Vol. 6, No. 8, pp. 1064– 1076, 1997. A. Zomet, A. Rav-Acha, and S. Peleg, “Robust super resolution, ” Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’ 01), Vol. 1, pp. 645 -650, 2001. D. G. Krige “Statistical approach to some basic mine valuation problems on the Witwatersrand, ” J. Chem. Metall. Min. Soc. South Africa, Vol. 52, No. 6, pp. 119 -139, 1951. 12

Contents • Super-resolution (SR)? • Components of traditional Super-resolution (SR) image reconstruction algorithms – Sub-pixel Registration Algorithms – Image Reconstruction Algorithms • Kriging • Conclusions , Future Work. 13

Why Kriging? • We need to “add” data in the regions of detail – Locally increasing the resolution – Interpolation • Kriging has been shown to outperform all other approximation methods, under certain circumstances – Ideal for non-uniform, sparsely sampled data – No assumptions made about regular sampling, number of data points, etc. – No assumptions about the type of data being approximated – When processing time is not a constraint. Kriging for Interpolation Estimate the spatial correlation between samples Construct an ideal model that best fits the calculated correlation Estimate new values using Kriging 14

Calculating the Semi-variance • • Semi-variance (SV) is a measure of the degree of spatial dependence between samples A variogram is a measure of how quickly things change on average – Directionally dependant – Measure of variance vs. distance • At d=0 the SV should be 0, and at larger distances the SV should increase Semi variance; intrinsic relationship between points. d is a set distance window N(d) is the number of points that distance apart zi is the data value of a local variable, taken at location i zi+h is another measurement taken h intervals away. Once the variogram data points are calculated, a parametric model is fit to the data. This model is then used to perform the kriging. 15

Parametric Models • Linear • Spherical • Exponential • Gaussian 16

Kriging uses points to estimate the value of an unknown value using a wieghted summation of nearby known points. Ideally, kriging attempts to choose the optimal weights that produce the minimum estimation error where the scatter of the estimates about the actual value has an estimation variance of • We want to find the optimal weights • Solving for W yields the weights necessary to perform kriging for a single estimate. These are found by solving a system of equations consisting of the weighted semivariances 17

SR Image Reconstruction using Kriging • • • We employ Kriging at the second stage of a standard SR image reconstruction method to increase the resolution of the imaging system. We applied this method to synthetic and real world data and compared the results with several other interpolation methods. The experiments show that our image reconstruction approach obtains better results than the competing algorithms. 18

Raw Image Sequence Image Registration SR Image Reconstruction 2 x Interpolation 2 x Optical zoom (ground truth) 2 x SR image (output) with Kriging 19

Experiments Keren+POCS Keren+Backprojection Keren+Bi-cubic Keren+Robust SR Keren+Kriging 20

Experiments Vandewalle+POCS Vandewalle+Back projection Vandewalle+Bi-cubic Vandewalle+Robust SR Vandewalle+Kriging 21

Experiments Raw Image Sequence Image Registration SR Image Reconstruction 2 x Interpolation 2 x Optical zoom (ground truth) 2 x SR image (output) with Kriging 22

Experiments Keren+POCS Keren+Backprojection Keren+Bi-cubic Keren+Robust SR Keren+Kriging 23

Experiments Vandewalle+POCS Vandewalle+Backprojection Vandewalle+Bi-cubic Vandewalle+Robust SR Vandewalle+Kriging 24

Experiments Keren et al. [3] Interp. Method Back Projec. Bi-cubic POCS Robust SR Kriging Mean St. Dev. 0. 93 5. 48 0. 97 9. 81 0. 75 5. 97 0. 60 3. 98 0. 32 2. 49 Vandewalle et al. [2] 1. 31 6. 91 1. 02 9. 89 0. 66 5. 34 0. 66 4. 28 0. 36 2. 68 Keren et al. [3] Var. 30. 07 96. 27 35. 69 15. 84 6. 21 RMSE 5. 56 9. 86 6. 02 4. 02 2. 51 Interp. Method Back Projec. Bi-cubic POCS Robust SR Kriging 47. 77 97. 99 28. 60 18. 35 7. 21 7. 03 9. 95 5. 38 4. 33 2. 71 Back Projec. Bi-cubic POCS Robust SR Kriging Performance evaluation of Kriging and other image reconstruction methods on license plate data Mean SD 13. 46 33. 76 9. 84 28. 88 14. 55 23. 86 9. 34 21. 21 9. 15 19. 74 Vandewalle et al. [2] 8. 97 20. 48 9. 99 28. 91 13. 45 23. 86 9. 34 21. 63 8. 23 18. 72 Var. 1139. 80 825. 09 569. 41 450. 27 426. 09 RMSE 36. 55 30. 51 27. 95 23. 11 20. 27 419. 77 836. 06 569. 41 468. 10 405. 09 22. 36 30. 59 27. 95 23. 56 20. 45 Performance evaluation of Kriging and other image reconstruction methods on dictionary paragraph data 25

Conclusions • Results related to this work was submitted to ICPR 2010. • In each of the experimental results given, our automated Kriging method outperformed the other competing methods in terms of accuracy. For the datasets shown here, the data are assumed to be free of blurring and geometrical deformations. Thus, we expect our method to be most useful when the need for accuracy is high and the computation time is not a restriction. The numbers for the standard deviation and variance values show that the errors obtained for our algorithm are lower than the ones obtained for the benchmark methods. • • • 26