High Frame Rate Up Conversion Ana Bertran Problem
High Frame Rate Up Conversion Ana Bertran
Problem Statement Original Frame (30 fps) Dwnsmpld Frame (5 fps) S&H Recovered Frame (30 fps) LI MCw. A Hypothesis: if we have acceleration MCw. A should provide better results X=a+b*sin(a*t), vx=b*a*cos(a*t), ax=-(b*a)2*sin(a*t)
Motivation Uses of frame rate up conversion: n Converting between standards (PAL to NTSC). Using MCw. A less critical since frame rates not too far from each other. n Low bit rate compression for video-confrncng, video-phone and video games. MCw. A becomes more critical (from 10 fps to 30 fps) Problems with traditional methods: n S&H motion looks jerky, not smooth, very choppy n Linear interpolation without motion: image looks blurry where motion has occurred, we will see ghosts due to avg. btwn. frames.
Motivation Original Linear Interpolation Sample & Hold
MCLI vs. MCw. A LMC ME Catalog Occsns AMC
ME For forward and backward MVs Overlapping BM 16 x 16 blcks, move by 8 ½ pixel MV zoom in Finer MV Selection • Top Left block use its MV • 1 st row blocks min SAD (L, current or next frame) • 1 st colmn blocks min SAD (T, current or next frame) • others min SAD (T, L, current or next frame) Take out illegal MVs
Cataloging Occlusions For covered pixels we need to use frame 2 as the predicted image, one whose blocks we serch for. For uncovered pixels we need to use frame 3 as the predicted image, one whose blocks we serch for. Need to track block from 3 to 2 Criteria: compare SADs depending which one minimum classify. 1 2 3 4
Cataloging Occlusions Results blue=uncov red=no occsn blue=uncov red=cov Green=no occsn
LMC No occlusions 3 cases Uncovered pixels For each block along motion trajectory: MVx_est=w*MVx_3_2 MVy_est=w*MVy_3_2 Motion will be non-integer: interpolate Assumed motion in x Depending on +ve/-ve x mtn, unvov/cov: Covered pixels 3 No occlusions 2 Occlusions
AMC To solve distancex=vx*t+0. 5*ax*t 2 you need two MVs but you can get more accurate motion trajectories by solving a LMS problem on 3 MVs. For each MV between two frames take the previous and next MVs to estimate the trajectory. Need to solve:
Frame rate up conversion a difficult problem Need to use true motion vector fields but block matching does poorly with this, specially across object boundaries and if the moving object is too small. Can´t cope with discontinuities in the velocity plane. Some motion can fall in between pixels Dealing with occlusions Motion can change dramatically between frames and we won´t realize it. (Critical vels) Not to talk about scene changes! – we no longer will have enough MVs to track motion trajectory.
Results/Demo Will have the demo in the website by the end of the week. For now you can see the demo for the other types of FR up-conversion
Current temporary results
Future Work Need better MV estimation of the true MVs n Implement ½ pixel resoltn or n Use hierarchical block matching to find true motion MVs or n Object based interpretation of the video to smooth out MVs n Use 3 D Recursive Search Block Matching Solve occlusion problem for both x and y movement
References Using Motion-Compensated Frame-Rate Conversion for the Correction of 3: 2 Pulldown Artifacts in lman, Video Sequences, Kevin Hilman, Hyuon Wook Park, and Yongmin Kim True-Motion Estimation with 3 -D Recursive Search Block Matching Gerard de Haan, Paul W. A. C. Biezen, Henk Huijgen and Olukayode A. Ojo Digital Video Standards Conversion in the presence of accelerated motion Amdrew J. Patti, M. Ibrahim Sezan and M. Murat Tekalp Framee Rate Up- Conversion using transmitted true motion vectors Yen-Kuang Chen, Anthony Vetro, Huifang Sun, and S. Y. Kung
- Slides: 15