Crossscale Predictive Dictionaries for Image and Video Restoration

Cross-scale Predictive Dictionaries for Image and Video Restoration (Paper ID: 2314) Vishwanath Saragadam, Aswin Sankaranarayanan, Xin Li 1

Compressive sensing Measurement matrix Signal of interest • Solving underdetermined linear system of equations • Relies on sparsity of signal • Orthogonal Matching Pursuit: Efficient recovery method 2

Orthogonal matching pursuit • 3

Accuracy increases with dictionary size Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, and S. K. Nayar. Video from a single coded exposure photograph using a learned over-complete dictionary. In IEEE Intl. Conf. Computer 4 Vision, 2011

A real example: High speed video Frames 1 - 36 Coded image Recovered video Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, and S. K. Nayar. Video from a single coded exposure photograph using a learned over-complete dictionary. In IEEE Intl. Conf. Computer 5 Vision, 2011

Need structured dictionaries • Need very large dictionaries for high accuracy • Computational complexity increases with larger number of dictionary elements Endow structure in dictionary to reduce search time 6

Structure across scales for visual signals Image Wavelet transform Sparse Multiscale 7

Wavelet zero tree 8 Wavelet transform Sparse Multiscale Predictive Parent coefficient zero child coefficients most likely zero Well known only for images Extend wavelet zero tree to dictionaries 8

CROSS-SCALE PREDICTIVE DICTIONARIES 9

Proposed signal model Downsample 0 Signal 0 0 0 Dictionaries Sparse representation 0 0 0 0 0 Zero tree structure of sparse coefficients 10

Signal model Child coefficients Parent coefficient Restrict support Use low resolution approximation to restrict high resolution approximation Child atoms Parent atom 11

Speedup • 12

Proposed signal model Downsample 0 Signal 0 0 0 Dictionaries Sparse representation Use OMP at each scale Zero tree of coefficients 0 0 0 0 0 Zero tree structure of sparse coefficients Zero tree OMP How do we learn these dictionaries? 13

Dictionary learning • Recollect: KSVD: -- Dictionary update: Rank 1 SVD -- Sparse approximation: OMP 14

Dictionary learning—results Data: 24 x 24 RGB patches (Parent atom) (Child atoms) 15

Dictionary learning—results Data: 24 x 24 RGB patches 16

Dictionary learning – results Data: 8 x 8 x 32 video patches 17

Dictionary learning – results Data: 8 x 8 x 32 video patches 18

Signal recovery • Upsampler 19

Application: Video compressive sensing 8 high speed video frames http: //high_speed_video. colostate. edu/ Coded image 20

Application: Video compressive sensing 21

Application: Video compressive sensing Original video Recovery using OMP Time: 3. 71 min SNR: 15. 79 d. B Recovery using zero tree OMP Time: 16. 5 s SNR: 17. 81 d. B http: //high_speed_video. colostate. edu/ 22

Video compressive sensing: Real data Coded image Recovered video using 10, 000 dictionary atoms Recovered video using zero tree OMP Time = 3 minutes Time = 45 s Thanks to Dengyu Liu and Yasunobu Hitomi for sharing the data they collected with us. 23

Accuracy plots for images Image denoising metrics Image inpainting with randomly deleted pixels. N/M represents the number of pixels recovered for each known pixel value. 24

Accuracy plots for videos Video denoising metrics Video compressive sensing. N/M represents the number of frames recovered from each coded image. 25

Downsample Summary 0 Signal 0 0 0 0 0 0 Zero tree structure of sparse Dictionaries Sparse representation coefficients Novel signal model inspired by wavelets zero tree Appealing for high dimensional signals like videos Questions? 26

THANK YOU 27

EXTRA SLIDES 28

Model comparison table Signal Class Images Videos N Nlow 8 x 8 4 x 4 24 x 12 x 3 3 8 x 8 x 16 4 x 4 x 8 8 x 8 x 32 4 x 4 x 16 8 x 8 x 16 4 x 4 x 8 Tlow Thigh Klow Khigh 64 1024 8 8 4. 10 20. 67 K-SVD Accuracy (d. B) 21. 98 512 8192 8 8 22. 6 19. 64 20. 57 512 512 8192 16384 16 16 15. 87 15. 80 23. 81 16. 89 22. 62 22. 75 20. 72 21. 84 24. 09 21. 36 23. 27 Model Speedup Accuracy (d. B) 29

OMP – Computational Requirements • Per-iteration costs – Least squares (Mx. K system) O (MN) O (MK 2) – Dominant O (MN + MK ) – Forming proxy – Finding closest atom • Total costs (K-iterations) 2 O (MNK + MK 3) 30

Zero tree OMP – algorithm 31

Zero tree OMP speed up computation • 32

Zero tree OMP speed up computation • 33