Future Video Coding Prof Dr Ing Jrn Ostermann

Future Video Coding Prof. Dr. -Ing. Jörn Ostermann Institut für Informationsverarbeitung

Video Coding Technology Improvements Infrastructure Device • Constant need for improved compression More devices, more pixels, more frames, more user-generated content 50% rate reduction in 10 years • Constant need for additional functionality High dynamic range, wide color gamut, head mounted displays, augmented reality, virtual reality Prof. Dr-Ing- Jörn Ostermann ostermann@tnt. uni-hannover. de 2

Video Coding New usages: Panorama mono Prof. Dr-Ing- Jörn Ostermann ostermann@tnt. uni-hannover. de 3

Video Coding New usages: Panorama stereo Prof. Dr-Ing- Jörn Ostermann ostermann@tnt. uni-hannover. de 4

Use Cases for Future Video Codec • Existing Markets Terrestrial, and Satellite broadcasting, cable services, managed IPTV services, surveillance • New Markets Over-the-top (OTT) services, User generated content, video conferencing, gaming • Emerging Markets Augmented and Virtual reality, multicast streaming/broadcast; media centric internet of things Prof. Dr-Ing- Jörn Ostermann ostermann@tnt. uni-hannover. de 5

Requirements on Future Video Codec • Source material 8 k, 0 Hz, 120 Hz, WCG, HDR, 10 bit Rectangular, panorama, projected / viewports Natural, synthetic Monoscopic, multiview • Network interface Streaming, error resilience, scalability, low delay • Emerging Markets Augmented and Virtual reality, multicast streaming/broadcast; media centric internet of things Prof. Dr-Ing- Jörn Ostermann ostermann@tnt. uni-hannover. de 6

Relevant Bodies • • • ISO/IEC JTC 1/SC 29/WG 11 Motion Picture Experts Group (MPEG) ITU-T Study Group 16 with Q 6/16 (VCEG) Common work between MPEG and VCEG MPEG-4 AVC (ITU-T H. 264, ISO/IEC 14496 -10) HEVC (ITU-T H. 265, ISO/IEC 23008 -2) MV-HEVC, 3 D-HEVC as extensions of HEVC • Joint Video Experts Team (JVET) Future Video Coding Prof. Dr-Ing- Jörn Ostermann ostermann@tnt. uni-hannover. de 7

JEM – where we are • Experimental open source software “Joint Exploration Model“ (JEM) – Intended to investigate potential for better compression beyond HEVC – Open source code available for download https: //jvet. hhi. fraunhofer. de/ – Documentation openly and freely available for download: http: //phenix. it-sudparis. eu/jvet/ – HEVC software plus additional or improved compression tools • General architecture similar to HEVC – Hybrid closed-loop, high level syntax, reference picture management, etc. – Improved/new compression tools Prof. Dr-Ing- Jörn Ostermann ostermann@tnt. uni-hannover. de 8

Overall: Hybrid Coding / Closed Loop Input Video Signal General Control Split into CTUs - Transform, Scaling & Quantization General Control Data Scaling & Inverse Transform Intra Prediction Data Intra-Picture Estimation Filter Control Intra-Picture Prediction Deblocking & SAO Filters Intra/Inter Selection Motion Compensation Motion Estimation Prof. Dr-Ing- Jörn Ostermann ostermann@tnt. uni-hannover. de Quantized Transform Coefficients Header Formatting & CABAC Coded Bitstream Filter Control Data Motion Data Decoded Picture Buffer Output Video Signal Source: Prof. Dr. -Ing. Jens Ohm 9

JEM Results compared to HEVC (HM) • • Measured average rate reduction for same PSNR (Bjontegaard delta method) Average encoder and decoder run time Random Access Over HM-16. 16 (sequential, gcc 6. 2) Y U V Enc. T Dec. T Class A 1 -27. 2% -33. 1% -35. 2% 10. 7 x 5. 6 x Class A 2 -36. 3% -45. 2% -38. 2% 7. 8 x 6 x Class B -27. 0% -36. 9% -31. 7% 8. 8 x 6. 3 x Class C -26. 3% -31. 2% -33. 6% 11. 3 x 7. 8 x Class D -26. 0% -30. 0% -31. 5% 11. 5 x 11. 7 x Overall -28. 5% -35. 4% -33. 9% 9. 9 x 7. 2 x Prof. Dr-Ing- Jörn Ostermann ostermann@tnt. uni-hannover. de Source: Prof. Dr-Ing. Jens Ohm 10

JEM – HEVC – AV 1 Basketball Drive (Random Access) 40 39 38 PSNR [d. B] 37 36 35 34 33 32 31 30 0 2000 4000 6000 8000 10000 Bit rate [kbps] AV 1 Prof. Dr-Ing- Jörn Ostermann ostermann@tnt. uni-hannover. de JEM 12000 14000 16000 18000 HM 11

Call for Evidence results • Joint Call for Evidence on Video Compression with Capability beyond HEVC (April 2017) Test cases with eight 4 K/SDR, five HD/HDR, and five 360 -degree video sequences 4 rate points each • Results of Joint Call for Evidence on Video Compression with Capability beyond HEVC (July 2017) Evaluates JEM anchors and other proposals against HEVC Main 10 Expert viewing tests with double-stimulus approach, allows grading on an MOS (mean opinion score) scale Prof. Dr-Ing- Jörn Ostermann ostermann@tnt. uni-hannover. de Source: Prof. Dr. -Ing. Jens Ohm 12

Call for Evidence results • Test cases SDR / UHD 4 K, MOS over rate [kbps "Daylight Road" "Campfire Party" Source: JVET-G 1004 Prof. Dr-Ing- Jörn Ostermann ostermann@tnt. uni-hannover. de Source: Prof. Dr. -Ing. Jens Ohm 13

Call for Evidence results • Test cases HDR / HD, MOS over rate [kbps] "Hurdles" "Cosmos" Source: JVET-G 1004 Prof. Dr-Ing- Jörn Ostermann ostermann@tnt. uni-hannover. de Source: Prof. Dr. -Ing. Jens Ohm 14

Call for Evidence results • Test cases 360° Video, dynamic viewports, MOS over rate [kbps] Source: JVET-G 1004 "Kite Flite" Prof. Dr-Ing- Jörn Ostermann ostermann@tnt. uni-hannover. de "Harbour" Source: Prof. Dr. -Ing. Jens Ohm 15

Next Steps • Joint Call for Proposals on Video Compression with Capability beyond HEVC Test cases with 4 K/SDR, HD/HDR, 360 -degree video 4 rate points each HM anchors and JEM anchors Formal subjective assessment in labs • Timeline: Cf. P October 2017 Submissions by February 2018, subjective testing March/April Evaluation of proposals: April 2018 Next generation standard by end of 2020 Prof. Dr-Ing- Jörn Ostermann ostermann@tnt. uni-hannover. de Source: Prof. Dr. -Ing. Jens Ohm 16

Conclusions • Future Video Coding will focus on: Rectangular 10 bit WCG HDR UHD video • Future Video Coding will enable: Support for panoramic video Edge-based rendering for interactivity, games • Future Video coding will deliver in 2020: Coding efficiency for UHD at 50% of the rate of HEVC for the same quality UHD at typically less than 4 MBit/s • MPEG continues to deliver compression technology beyond the state of the art Prof. Dr-Ing- Jörn Ostermann ostermann@tnt. uni-hannover. de 17