Image and Video Upscaling from Local Self Examples

Image and Video Upscaling from Local Self Examples Gilad Freedman Raanan Fattal Hebrew University of Jerusalem

Single image upscaling 1. large 2. realistic 3. faithful 4. fast

Previous work parametric image model example based noisy result Freeman et al. 2002 Fattal 2007 Shan et al 2008 Sun et al. 2008 generic looking edges Glasner et al. 2009

New approach: locale example based local self similarity corner step edge local self similarity small upscaling ratios 4/5 local novel components: speed Increase exemplar search line step edge quality and size 1/2 1. local self similarity maintain search locality too few locality non smooth shading 2. examples non-dyadic filter bank new non-dyadic filter bank

Local self-examples upscaling frequency content interpolated image low pass original image high pass

Local self-examples upscaling For each patch: frequency content Search a local area for best example low pass Add to interpolated image Take corresponding patch high pass

Local self-examples upscaling Repeat for all patches, to fill the high frequencies frequency content interpolated image low pass high pass

Local self similarity �

Local self similarity Patches in original image can matched locally with ones in downscaled version

Local examples are enough query db full image local 4. 0 2. 9 1. 6 1. 05 2. 7 2. 05 3. 3 image database image 6. 5 3. 55 2. 96 5. 61 3. 06 5. 61 best matches

Visual assessment – external, exact NN, local Large external example database external database Searching the entire image global search Searching local regions in image local search

Comparison of example search methods

Need for non-dyadic scalings large ratios small ratios mixed ratios

Dyadic filters 1: 2 full frequency content higher half lower half dyadic filter bank

Non-dyadic filter bank 4: 5 full frequency content 1: 2 small scaling ratios better examples higher part lower part non-dyadic filter bank

Non-dyadic filters: downscaling example for the 2: 3 ratio: dyadic case: 1. convolve with one filter 2. subsample by 2 1. convolve with 2 filters 2. subsample each by 3

Non-dyadic filters: upscaling example for the 2: 3 ratio: dyadic case: 1. zero upsample by 1 2. convolve with 1 filter 1. zero upsample by 2 2. convolve with 2 filters 3. sum

Use of the filters in upscaling Upscaling using inverse scaling filters Smoothing by downscaling and upscaling

1. Uniform stretch brightness When interpolating, smooth areas come from input Uniformly spaced grids should remain uniform 255 0 grid coordinates

2. Consistency upsample downsample Previous methods achieve consistency by solving large linear systems to achieve this property

3. PSF modeling Difference between point spread functions Large image - small camera point spread function Small image - large camera point spread function

frequency 4. Low frequency span When upsampling don’t add new frequencies original Upsampling filter should be low-pass interpolated

5. Singularities preservation similar amount of blurred Image interpolated image

Real time video upsampling on GPU Search and filter-banks are both local operations main GPU memory GPU cores NTSC to full HD @ 24 fps

Bicubic x 3 (zoomed in) Ours X 3 (zoomed in)

Bicubic x 3 Ours X 3

Genuine Fractals™ x 4 Ours X 4

Glasner et al. 2009 x 4 Ours X 4

Thank you! Paper & additional results can be found at: www. cs. huji. ac. il/~giladfreedmn