ContextConstrained Hallucination for Image SuperResolution ChihYuan Yang Electrical

Context-Constrained Hallucination for Image Super-Resolution Chih-Yuan Yang Electrical Engineering and Computer Science October 11 2011

Overview • • • Introduction Existing algorithms Idea Results Conclusion 2

Problem Definition • Two perspectives – Reversion – Creation 3

Reversion • The ground truth exists • The downsampling function is given • Metrics of error can be defined Inverse Function 128 x 128 512 x 512 4

Creation • No ground truth • Evaluated in visual perception 1024 x 1024 Creation 512 x 512 5

Overview • • • Introduction Existing algorithms Idea Results Conclusion 6

Approaches • Interpolation – fast and simple – poor quality • Learning – Statistics-based • low computational low, no user guidance • less flexibility – Example-based • flexibility • high-computational load, user guidance (some), restricted (some) 7

Existing Algorithms Approach Statistics Content / Example Source Bicubic Interp. non-learning Back Projection non-learning Yang CVPR 08 statistics high-frequency patch Sun CVPR 08 statistics gradient profile Tai CVPR 10 stat. + ex. gradient profile + one exemplar image Shan SIGGRAPH 08 statistics gradient distribution Glasner ICCV 09 example self example Freedman TOG 10 example self example Ha. Cohen ICCP 10 example small dataset of texture images Sun CVPR 10 stat. + ex. large dataset of natural images 8

Overview • • • Introduction Existing algorithms Idea Results Conclusion 9

1. Segment Search Query (Bicubic Interpolation) Patch Search Segment Search Ground Truth 10

Example Query Segments Retrieved Segments 11

Implementation Details • Segment algorithm: Berkeley Segmentation Engine • Segment feature: Histogram of 1 st order derivative. Feature length 360, from 3 scales, 6 orientations, 20 bins. • Similarity metric: 12

High Frequency Detail Mapping • Patch search: minimal MSE • Patch reproduction 13

2. Edge Enhancement 14

Formulation • Reconstruction • Hallucination • Edge 15

Gradient Descent • Reconstruction • Hallucination • Edge 16

Overview • • • Introduction Existing algorithms Idea Results Conclusion 17

Doll Shan 08 (Gradient Distribution) Sun 08 (Gradient Profile) Sun 10 (Segment Search) 18

Zebra Irani 93 (Back Projection) Glasner 09 (Self Similarity) Sun 10 (Segment Search) 19

Koala Glasner 09 (Self Similarity) Sun 10 (Segment Search) Ho. Cohen 11 (Texture Synthesis) 20

Child Fattal 07 (Edge statistics) Glasner 09 (Self Similarity) Sun 10 (Segment Search) Ho. Cohen 11 (Texture Synthesis) 21

My Implementation Example Image 22

Segmentation Results 23

Search Results 24

Edge 25

Results small lambda medium lambda large beta 2 26

Overview • • • Introduction Existing algorithms Idea Results Conclusion 27

Conclusion • Texture – Exploit texture similarity • Edge – Edge enhancement 28

Characteristics Main idea Image Quality (Style) Computational load User Guidance Bicubic Interp. interpolation over-smooth extremely low no Back Projection minimal intensity difference ringing artifacts extremely low no Glasner ICCV 09 cross-scale self-patchsimilarity good edges high no Freedman TOG 10 local cross-scale self-patchsimilarity good edges, triangle artifacts in texture regions medium no Yang CVPR 08 high-band patch statistics full of details, edge artifacts low no Sun CVPR 08 edge statistics good edges medium no Tai CVPR 10 gradient copy full of details, noisy high yes Shan SIGGRAPH 08 gradient statistics blocky low no Ha. Cohen ICCP 10 texture synthesis vivid texture, unlike the input Sun CVPR 10 segment matching, patch transfering full of details yes medium no 29