MPEG for the past present and future of

MPEG for the past, present and future of television Leonardo Chiariglione Keynote Speech at ATSC Annual Meeting Washington, DC – 2012/05/08

MPEG for the past of television System Transport Stream DSM-CC Timed metadata Video MPEG-2 High Profile Audio MPEG-1 Audio Layer II MPEG-2 AAC MPEG for the past, present and future of TV 2012/05/02 2

An assessment MPEG has provided the means for the television business on air, cable and satellite to migrate from the analogue to the digital age It has been a collective efforts Representatives from the entire spectrum of the broadcasting world provided their requirements Packages of patent licences were created (outside of MPEG) Today there is virtually no broadcasting system that is not based, at least partially, on MPEG standards MPEG for the past, present and future of TV 2012/05/08 3

MPEG for the present of television Systems Carriage of MPEG-4 on MPEG-2 TS Dynamic Adaptive Streaming over HTTP (DASH) Video MPEG-4 Advanced Video Coding (AVC) 3 D extensions Audio Various forms of Advanced Audio Coding (AAC) Universal Speech and Audio (USAC) MPEG for the past, present and future of TV 2012/05/08 4

An assessment The deployment of 2 nd generation digital broadcasting systems continues successfully Use of the AVC on MPEG-2 TS allows packing twice as many programs as MPEG-2 Video in an analogue channel With IPTV the notion of broadcasting is expanding MPEG is helping the broadcasting industry To enrich the broadcasted user experience Through standard technologies that serve multiple fields MPEG for the past, present and future of TV 2012/05/08 5

MPEG for the future of television Well, that’s what I am supposed to talk about… MPEG for the past, present and future of TV 2012/05/08 6

Broadcasting is a great business to be in… Broadcasting is a proven business with a constant revenue flow Many technologies appear that Can help new competing businesses but Can also be used to extend the broadcasting business MPEG is a neutral body devoted to the development of standard technologies for user communities to exploit MPEG for the past, present and future of TV 2012/05/08 7

MPEG-H: a video standard for future broadcasting ISO/IEC 23008 High Efficiency Coding and Media Delivery in Heterogeneous Environments A suite of standards for media coding and delivery Currently scheduled to appear in 3 parts MPEG Media Transport High Efficiency Video Coding 3 D Audio Timeline Part CD DIS FDIS MMT 12/07 13/01 13/07 HEVC 12/02 12/07 13/01 3 D Audio 14/04 14/10 15/01 MPEG for the past, present and future of TV 2012/05/08 8

Service composition in MMT Package composition information friendly with html 5 <subtitle> Bird is flying… UD contents 3 D Earth Bird is flying… legacy broadcasting MPEG for the past, present and future of TV future broadcasting 2012/05/02 9

Smart delivery in Clouds video Web pull file Clouds pull MMT Encapsulator - Service Component List Component location Composition relationship Qo. E requirement How to deliver How to consume MMT Frame - Configuration - Adaptive delivery - Sync, Delay, Loss control 2012/05/08 10

Mash-up service Audio Video 1 Video 2 widget Web page MMT assets Service 1 (MMT Package 1) MPEG for the past, present and future of TV Service 2 (MMT Package 2) 2012/05/08 11

Second screen management MMT Package composition information <device 1> … <device 2> … Video content on 2 nd device Additional Information with Live Content (HTML) MPEG for the past, present and future of TV 2012/05/08 12

MPEG-H part 2 High Efficiency Video Coding Primary goal: ≥ 50% better compression than AVC For large displays (e. g. 4 kx 2 k) For mobile (e. g. WVGA) Developed jointly with ITU-T by Joint Collaborative Team (JCT) on Video Coding Technically in the same stream of past MPEG Video Coding standards (AVC and prior standards) Currently at Committee Draft level MPEG for the past, present and future of TV 2012/05/08 13

HEVC vs AVC Subjective Performance Measurements Preliminary subjective tests carried out for HEVC Committee Draft Reasonably small and non-overlapping confidence intervals Average rate savings at same (interpolated) MOS points HEVC with 67% lower rate in class B (Full HD) HEVC with 49% lower rate in class C (WVGA) HEVC with 58% lower rate overall MPEG for the past, present and future of TV 2012/05/08 14

A performance sample MPEG for the past, present and future of TV 2012/05/08 15

For those technology aware… Block-based/Variable block sizes Block motion compensation Fractional-pel motion vectors Spatial intra prediction Spatial transform of residual difference Integer-based transform designs Arithmetic or VLC-based entropy coding In-loop filtering to form final decoded picture MPEG for the past, present and future of TV 2012/05/08 16

Where we are (May 2012) Activity: level of 1, 000 documents per meeting crossed Very diverse participation from industry/academia Significant technical advance over prior standards Challenge: computational/implementation complexity Deliverables: Spec Reference software Conformance Currently one profile MPEG for the past, present and future of TV 2012/05/08 17

HEVC Scalability Extensions/1 Serve the needs of heterogeneous environment of future digital TV distribution Multiple devices: SHDTVs, set-top boxes, tablets, smart phones, PCs etc. Wide range of processing powers, display sizes and power consumption needs Multiple networks: Cable, Satellite, Terrestrial, CDN, Cable Modem, x. DSL, Wi. Fi, 4 G Wide range of channel bandwidths and Qo. S Multiple protocols: MPEG-2 TS, DASH, HLS, RTP, UDP Multiple services: Broadcasting, On demand, Streaming, Over The Top MPEG for the past, present and future of TV 2012/05/08 18

HEVC Scalability Extensions/2 Options for backward compatible migration paths From 720 p 60 to 1080 p 60 From 1080 p 60 to 4 kx 2 kx 60 From AVC to HEVC Time Line May 2012 Preliminary Call for Proposals (public) July 2012 Final Call for Proposals (public) October 2012 Evaluation of proposals & start of collaborative design phase Development of the time line associated with standardisation phases (WD, CD/PDAM, FDIS/FDAM, DIS/AMD) MPEG for the past, present and future of TV 2012/05/08 19

Service Transitions 1950 s: Color TV Analog, backward compatible 1990 s: Digital TV New infrastructure required Transitions from SD to HD 2010 s: 3 D Mixed results from services introduction is Not a single format across all services MPEG for the past, present and future of TV 2012/05/08 20

Current Status of 3 D Video Increasing investments in 3 D production and services Increasingly capable 3 D displays in the market (many competing and emerging technologies) Market is primarily stereo (mix of different formats being deployed right now) Focus of current MPEG work: development of a new 3 D format and associated compression techniques that could facilitate generation of multiview output MPEG for the past, present and future of TV 2012/05/08 21

Existing 3 DV Coding Formats Full-resolution coding of stereo and Left Right MV video as extension of AVC High coding efficiency achieved via inter-view prediction techniques 2 D compatibility supported Adopted as format for 3 D Blu-ray Disc Being considered for second phase of broadcast standards Pack pixels from left and right views into a single frame (loss of resolution) Leverage existing infrastructure and equipment, only require additional signaling to de-interleave frame Embraced by broadcasters for first phase of 3 D broadcast services MPEG for the past, present and future of TV 2012/05/08 22

Targets of Future 3 DV Format MPEG for the past, present and future of TV 2012/05/08 23

3 DV Reference Framework Limited # Video Inputs (e. g. , 2 or 3 views) Depth Estimation Video/Depth Encoder Decoder and View Synthesis Larger # Output Views 1010001 + Binary Representation & Reconstruction Process MPEG for the past, present and future of TV 2012/05/08 24

Overview of Call For Proposals Test Material & Conditions 8 test sequences (1920 x 1080 & 1024 x 768) 4 target bit rates per sequence 2 -view and 3 -view test scenarios AVC and HEVC compatible test categories 23 responses (12 AVC + 11 HEVC) Evaluation in Nov 2011 Objective quality (PSNR of synthesized views) Subjective assessment (stereo and auto-stereo) MPEG for the past, present and future of TV 2012/05/08 25

3 DV Testing Scenarios 2 -view Configuration 3 -view Configuration MPEG for the past, present and future of TV 2012/05/08 26

Tool Categories Texture coding Independent of depth E. g. , inter-view prediction of color view, inter-view prediction of motion parameters and residual data Using depth data E. g. , view synthesis prediction, motion prediction Depth coding Independent of texture E. g. , depth modeling modes, weighted prediction, reduced resolution coding Using texture data E. g. , motion parameter inheritance, intra prediction MPEG for the past, present and future of TV 2012/05/08 27

3 DV Current Status Extensive subjective evaluation of 20+ proposals to the 3 DV Cf. P Considered both AVC and HEVC compatible Many new coding tools proposed for MV texture and depth coding Substantial rate savings compared to capabilities of existing standards AVC and HEVC extensions standardisation under wayd Add support for depth New AVC-based coding tools MV extensions of HEVC Hybrid architectures MPEG for the past, present and future of TV 2012/05/08 28

Hybrid Architectures From a pure compression efficiency point of view, it is always best to use the most advanced codec However, when introducing new services, providers must also consider capabilities of existing receivers and transition plan Use Case: Many terrestrial broadcast systems based on MPEG-2 US cable systems based on mix of MPEG-2 and AVC May not be easy to simply switch codecs in near term MPEG for the past, present and future of TV 2012/05/08 29

Multiview Extensions of HEVC Stereo/MV extensions being considered Extensions expected to be completed approximately 12 months after the base specification, i. e. , Jan 2014 Reportedly simple extensions of HEVC can achieve 3040% bit rate reduction compared to HEVC simulcast Further gains expected if block-level changes to codec are considered MPEG for the past, present and future of TV 2012/05/08 30

MPEG-H part 3: 3 D Audio Well, there is no Systems and Video without Audio… MPEG for the past, present and future of TV 2012/05/08 31

3 D Audio requirements/1 High quality: quality of decoded sound perceptually transparent Localization and Envelopment: Accurate sound localization and very high sense of sound envelopment within a targeted listening area (“sweet spot”) Flexible Loudspeaker Placement: the transmitted audio program to be mapped to a rendering setup in which loudspeakers are in alternate locations Interactivity: Interactive modification of the sound scene rendered from the coded representation MPEG for the past, present and future of TV 2012/05/08 32

3 D Audio requirements/2 Rendering on setups with fewer loudspeakers (including headphones): ability to derive a signal from the transmitted program material for reproduction with fewer loudspeakers Audio/visual alignment & consistency: ability to adapt the rendered acoustic scene to be consistent with the visual Efficiency for decoding on different setups: ability to be rendered on all reproduction loudspeaker setups or headphone MPEG for the past, present and future of TV 2012/05/08 33

3 D Audio requirements/3 Appropriate computational complexity and system latency for the target application (Broadcasting, Spatial two-(n-)way communication / telepresence Transcoding for low bandwidth devices: A lower bandwidth signal can be extracted from the original program material Backward compatibility: (e. g. to 5. 1 channel programs and decoders or transmission of HRTF encoded signals) MPEG for the past, present and future of TV 2012/05/08 34

Envisioned Architecture Inputs • Audio Channels “normal” content Audio Objects “helicopter overflight” Channel – 500 kb/s primary channel – 64 kb/s cellular channel • Outputs – Loudspeakers – Headphones • Spatialized sound – In-frame loudspeakers • “Sound Bar” Audio Scene Ambisonics MPEG for the past, present and future of TV 2012/05/08 35

Audio, part of an Audio-visual Experience What is new in video? Larger, high-resolution displays Greater sense of envelopment Possibly closer viewing distance More efficient compression Wireless transmission to portable devices Immersive and enveloping audio experience MPEG for the past, present and future of TV 2012/05/08 36

Use Case 1 Home Theatre Many loudspeakers 10. 1 22. 2 “ 3 D” Experience Height loudspeakers for greater envelopment Sense that audio objects are near or distant When near, they are consistent with video image MPEG for the past, present and future of TV 2012/05/08 37

22. 2 multichannel sound system (NHK) MPEG for the past, present and future of TV 2012/05/08 38

Home Theater – Issues Maintain compatibility with legacy systems: 5. 1, stereo Will content providers really step up to 22. 2? Will consumers adopt 22. 2? : 2 front high How to compress 22. 2 for transmission? MPEG AAC 64 kb/s/channel: -> 1. 5 mb/s How to render for legacy setups? Relatively slow-growth market MPEG for the past, present and future of TV 2012/05/08 39

Use Case 2 Personal TV Small but high-resolution display “super-tablet” Local wireless communications To cable or fiber home hub Possible audio presentation Headphones Loudspeakers around perimeter of display Issue: To what extent can the user get an enveloping experience from only “front” loudspeakers? MPEG for the past, present and future of TV 2012/05/08 40

Use Case 3 Mobile TV Hand-held display: Smartphone Headphone listening: Stereo, perhaps with binauralisation Fastest growing market: Quick rollout and adoption of standard What is new? USAC for stereo; MPEG Surround for binauralisation What is the “Wow” factor? To spur adoption of new technology MPEG for the past, present and future of TV 2012/05/08 41

User Experience Spatialized on headphones Home Theatre Tablet TV MPEG for the past, present and future of TV 2012/05/08 42

Flexible Rendering Content providers create 22. 2 program Consumed on stereo, 5. 1, 10. 1, 22. 2 layouts Consumed on “wrong” layouts Mis-positioned surrounds Missing surrounds Non-standard layouts 2 front high, 5. 1 mid MPEG for the past, present and future of TV 2012/05/08 43

3 D Audio Test Platform NHK “Loudspeaker Array Frame” (LAF) Suggestions for Alternative platform for assessing “Home Theatre” use case Model for assessing “Personal TV” use case MPEG for the past, present and future of TV 2012/05/02 44

Summary Provide compelling immersive audio experience for audio/visual programs Create content once, present on many different loudspeaker layouts or on headphones Provide high compression and high-quality audio presentation MPEG for the past, present and future of TV 2012/05/08 45

MPEG-V: Humans have more than two senses MPEG for the past, present and future of TV 2012/05/08 46

Augmented Reality Application Format (ARAF) Remote Real World Environment Remote Sensors & Actuators Local Real World Environment Local Sensors & Actuators MARBle User ARAF file or stream Media Servers MPEG for the past, present and future of TV Service Servers 2012/05/08 47

Augmentation Region in ARAF AR service provider A User A Broadcaster User B Augmentation Region MPEG for the past, present and future of TV AR service provider B 2012/05/02 48

The past, the present and the future, again 25 years ago MPEG selected the digital media way bringing innovation and interoperability …and thrived This year the broadcasting industry has started a move in future broadcast services with interoperability at its core …and it is bound to thrive MPEG looks forward to continuing a cooperation with the broadcasting industry providing the necessary standard technologies MPEG for the past, present and future of TV 2012/05/08 49

http: //mpeg. chiariglione. org/ MPEG for the past, present and future of TV 2012/05/08 50