Viewportbased 360 Video Streaming MotionConstrained Tile and Viewport

Viewport-based 360 Video Streaming: Motion-Constrained Tile and Viewport Extraction Presenter: Eun-Seok Ryu (esryu@gachon. ac. kr) Dongmin Jang, Jong-Beom Jeong, Eun-Seok Ryu Multimedia Communications and Systems Lab. (MCSL) http: //mcsl. gachon. ac. kr Department of Computer Engineering Gachon University

Test Materials for 3 Do. F+ - w 17726 v Philips § Classroom. Video v Technicolor § Technicolor. Museum § Technicolor. Hijack Class A Classroom. Video (4096 x 2048), 360º x 180º FOV ERP, 30 fps, 120 frames, 15 source views 3079 -18 -0046 -00 -0002 Class B Technicolor. Museum (2048 x 2048), 180º x 180º FOV ERP, 30 fps, 300 frames, 24 source views Class C Technicolor. Hijack (4096 x 4096), 180º x 180º FOV ERP, 30 fps, 300 frames, 10 source views

Common Test Conditions (CTC) for 3 Do. F+ - w 17726 v Software name Location Tag/branch RVS http: //mpegx. int-evry. fr/software/MPEG/Explorations/3 Do. Fplus/RVS v 2. 0 ERP-WS-PSNR http: //mpegx. int-evry. fr/software/MPEG/Explorations/3 Do. Fplus/ERP_WS-PSNR 20180704 HDRTools https: //gitlab. com/standards/HDRTools/tree/0. 18 -dev 360 Lib https: //jvet. hhi. fraunhofer. de/svn_360 Lib/branches/360 Lib-5. 1 -dev HM 3079 -18 -0046 -00 -0002 https: //hevc. hhi. fraunhofer. de/svn_HEVCSoftware/tags/HM-16. 16

Common Test Conditions (CTC) for 3 Do. F+ - w 17726 v Anchor definition Definition of the anchor Technical proposal with pre- and post-processing 3079 -18 -0046 -00 -0002

Common Test Conditions (CTC) for 3 Do. F+ - w 17726 v Quantization Parameter and class QP 1 QP 2 QP 3 QP 4 QP 5 Depth QP 12 17 22 27 32 Texture QP 22 27 32 37 42 QPs used for depth and texture Test class Sequence Name No. of source No. of anchorviews coded views Anchor-coded views A 1 Classroom. Video 15 15 All A 2 Classroom. Video 15 9 v 0, v 7…v 14 B 1 Technicolor. Museum 24 24 All B 2 Technicolor. Museum 24 8 0, 1, 4, 8, 11, 12, 13, 17 C 1 Technicolor. Hijack 10 10 All C 2 Technicolor. Hijack 10 5 1, 4, 5, 8, 9 Anchor-coded views per class 3079 -18 -0046 -00 -0002

Common Test Conditions (CTC) for 3 Do. F+ - w 17726 v 3 Do. F+ objective evaluation Term Description Coded view position The position of a source view which is coded by the anchor Non coded source view position The position of a source view which isn’t coded by the anchor Intermediate view position The position of a view which is out of any source view Definitions for test class on 3 Do. F+ WS-PSNR For the anchor RD curve For the proponent’s RD curve Coded BD rate All frames Coded view & source view All frames Proponent’s view & source view Non-coded BD rate All frames Non-coded source view & source view All frames Proponent’s view & source view Intermediate BD rate All frames Intermediate view made by decoded views & same view made by all source views with 16 -bit depth maps All frames Proponent’s intermediate view & Same view made by all source views with 16 -bit depth maps Requirements for 3 Do. F+ objective evaluation 3079 -18 -0046 -00 -0002

Reference View Synthesizer - w 17759 v Reference software for view synthesis of 3 Do. F+ Conceptual Diagram Simple Example Virtual View Blending by dividing High Frequency and Low Frequency Virtual View Blending View & Depth Synthesis using Warping Texture & Depth Input View 1 Input View 2 Input View 3 Input View 4 3079 -18 -0046 -00 -0002 … … View & Depth Synthesis Input View L Input View R

Reference View Synthesizer - w 17759 v 3 D Warping § § Pixel movement between viewpoints of the same world coordinate Input view is divided into triangles with the pixels centers as vertices Use affine transformation Being filled with interpolated colors after affine transformation (a) Input view 3079 -18 -0046 -00 -0002 (b) Obtained view after synthesizing view and depth

Reference View Synthesizer - w 17759 v View synthesis M Camera World Considering principal point Synthesis view Input view Fig 1. Camera and world cordinates of input and synthesis views Optical axis Camera coordinate move by t and rotate by R from world coordinate u f Image plane Fig 2. Camera coordinate above the image plane 3079 -18 -0046 -00 -0002

Reference View Synthesizer - w 17759 v. Good quality : low depth and triangle with a regular shape v. Taking the pixel with the maximal quality would give a sharper result v. Taking the weighted mean is more resistant to errors v. High and low frequencies are separated with a mean blur v. Low frequencies are blended with the weighted mean v. High frequencies are blended by choosing the pixel of highest weight depth at pixel for synthesized view i (a ) (a): Blending by argmax (b ) (c ) (b): Weighted mean (c): Multi-spectral blending: argmax - high frequencies weighted mean - low frequencies 3079 -18 -0046 -00 -0002

Reference View Synthesizer - w 17759 v Architecture Pipeline Object (Pipeline. cpp) Initialize Process Parse Configurations Routine: Load Image per Frame Pipeline: : compute_views(int Frame) Routine: Create Virtual View(Renderd View) to Write, For i = 1 to num_virtual_view Set Blending Method Type: 1)Simple 2)Multi. Spec CTC use (1) Set Virtual Camera Parameters RVS define as Projector Initialize Image Buffer Inpainting Type: 1)Interpolation 2)Nearest RVS only uses (2) Using Manhattan Not Main Interest Downscale Write Output. File & Out. Masked. File Downscale rasterized image Routine: Blend Virtual Views from Input Views, For i = 1 to num_input_view Set Image Params Compute Views 3079 -18 -0046 -00 -0002 Set Projection Type Set Input Camera Parameters Set Synthesis Method Type: 1)Perspective 2)Equirectangular RVS define as Un. Projector RVS only has Triangle Method Load Input View Image Synthesis Blending

Fast Color Correction for View Synthesis - m 43694 v. Fast color correction technique for view synthesis v. Reduction of color artifacts in synthesized images § Offset calculation § Offset addition SVS, no color correction (31. 7 d. B) SVS, with color correction (32. 6 d. B) PSNR [d. B] Test sequence View Synthesis with Proposed Color Correction (oc* – offset calculation) 3079 -18 -0046 -00 -0002 Ballet Breakdancers Poznan_Fencing 2 Soccer Linear BBB Butterfly BBB Flowers VSRS No color With color correction 31. 45 31. 96 31. 89 26. 14 28. 04 34. 80 32. 06 22. 71 SVS No color With color correction 31. 72 32. 55 32. 09 32. 16 26. 79 28. 50 34. 92 34. 94 33. 91 22. 51 22. 48

3 Do. F+ Software Platform Description - w 17761 v 3 Do. F+ Software platform 3079 -18 -0046 -00 -0002

Outperforming 3 Do. F+ Anchors – m 43504 v Removing overlap between anchors reduces data v Send a central view and sparse views 3079 -18 -0046 -00 -0002

Outperforming 3 Do. F+ Anchors – m 43504 v 1 central ERP + 24 semi ERP views 3079 -18 -0046 -00 -0002

Source View Pruning & Packing v Based on m 43504, build source view pruning module with RVS v Packing module divides the sparse views into small size blocks v If the block contains information more than threshold, it is added to Central View packing view Packing Input View Source View Pruning Sparse View Divide view into blocks Remove redundancy Pixel-by-pixel comparision v 14 s 1 Input View … … v 1 Central View s 14 3079 -18 -0046 -00 -0002 Add blocks which contain information more than threshold

Source View Pruning & Reconstruction v Architecture Source views Sequences in Source Format 3079 -18 -0046 -00 -0002 Anchor-coded views Central & Source View Selection Source View Pruning with RVS Bitstream Decoded views HEVC Encoding HEVC Decoding Source View Reconstruction Measure WS-PSNR with Source Views

Source View Pruning & Reconstruction v RD-curve 3079 -18 -0046 -00 -0002

Conclusion v Motivation § 3 Do. F+ requires high resolution, large amount of videos § Multi-view video transmission needs multiple decoders v Proposed method § Generate sparse view by removing overlap between views with RVS § Divide sparse view into blocks and add them to packing view § Reconstruct source view with central view and sparse views v Results § Source view pruning saves bitrate with small loss of PSNR § Packing reduces the size of view to transmit v Future work § Color correction technique with illumination compensation § Extensive experiment for optimal parameters 3079 -18 -0046 -00 -0002