Learning StructureAndMotionAware Rolling Shutter Correction Bingbing Zhuang QuocHuy

Learning Structure-And-Motion-Aware Rolling Shutter Correction Bingbing Zhuang, Quoc-Huy Tran, Pan Ji, Loong Fah Cheong, Manmohan Chandraker

Rolling shutter cameras are everywhere …

Rolling Shutter Effect Sequential Exposure Mode 3 D Point Camera motion between scanlines

Rolling Shutter Distortion Our Correction

Difficulties • Complexity of RS geometry Two-view approach [Dai et. al. CVPR 17], [Zhuang et. al. ICCV 17] • Degeneracy of RS geometry q RS two-view Sf. M under pure translation: a critical configuration (contribution 1) q Our single-view approach (contribution 2) ü Data-driven with CNN ü Structure-And-Motion-Aware

Critical Configuration Analysis Proposition. RS two-view Sf. M under pure translational camera motion is degenerate. Ø Infinitely many solutions exist for per-scanline camera positions and depth. See paper for a proof Radiating Pattern

Our Single-view Approach Network Architecture Rectification RS GS Camera motion between scanlines

Training Data Generation Synthesize: GS RS 3 D points RS image GS depth Camera motion between scanlines RS depth

An Ambiguity In Training Data Preprocessing Different sizes Same size Resize?

An Ambiguity In Training Data Preprocessing VS Vertical Resizing Similar to the Bas-Relief ambiguity in classical Sf. M

Result: Synthetic RS Image RS Ground Truth GS Ground Truth Warping Flow

Result: Synthetic RS Image GT Ours 2 DCNN MH (1) MH, [Purkait et. al. ICCV 17] (2) 2 DCNN, [Rengarajan et. al. CVPR 17] (3) Ours

Result: Real RS image Input MH 2 DCNN Ours

Conclusion • RS geometry may suffer from degeneracy. • Geometrical guidance is the key to improving RS correction. Rectification • Future work: Motion deblurring For more details, visit www. neclabs. com/research-departments/media-analytics/.