Overview of the MPEG4 Version 1 Standard 12

Overview of the MPEG-4 Version 1 Standard (1~2) 指導老師: 龔旭陽 教授 報告人: 林郁廷 2020/11/27 1

Table of Contents 1. Scope and features of the MPEG-4 standard 2. Detailed technical description of the MPEG-4 standard 3. List of major functionalities provided by MPEG-4 in January ’ 99 4. Profiles in MPEG-4 Version 1 5. Annexes 2020/11/27 2

1. Scope and features of the MPEG-4 standard 1. 1 Representation of primitive AVOs 1. 2 Composition of AVOs 1. 3 Multiplexing and Synchronization of AVOs 1. 4 Interaction with AVOs 1. 5 Identification and Protection of Intellectual Property Rights of AVOs 2020/11/27 3

1. 1 Representation of primitive AVOs • Audiovisual scenes are composed of several AVOs, organized in a hierarchical fashion. • At the leaves of the hierarchy, we find primitive AVOs, such as : 1. a 2 -dimensional fixed background, 2. the picture of a talking person (without the background) 3. the voice associated with that person; 4. etc. 2020/11/27 6

Add figure : 144996 terminal architecture 2020/11/27 7

Add Figure : The buffer process of the MPEG-4 system 2020/11/27 8

Add figure : Sync Layer recombine SL packages into access unit in according to timing information of each SL packet. 2020/11/27 9

Figure 1 - An example of an MPEG-4 audiovisual scene 2020/11/27 11

1. 3 Multiplexing and Synchronization of AVOs • 由一或多個基本串流傳遞 AV 物件資料。 • Qo. S 特性的串流需求傳輸如 maximum bit rate , bit error rate , etc. • 和其他參數一樣 , 串流型態資訊決定需求解碼器資源和 編碼時間資訊。 • in terms of an Access Unit Layer and a conceptual two-layer muliplexer • Access Units (e. g. , video or audio frames , scene description commands) 2020/11/27 12

Figure 2 - The MPEG-4 System Layer Model 2020/11/27 13

Add figure: MPEG-4 Terminal Architecture 2020/11/27 14

傳輸層 [con. . ] • DMIF 應用程式界面 (DMIF Application Interface ; DAI) 是位在同步層和 Flex. Mux 之間的 Session layer services。 2020/11/27 16

1. 5 Identification and Protection of Intellectual Property Rights of AVOs • 唯一的辨識碼為國際的數位系統採用。 • 用以辨識現行的所有權。 • Version 2 將陳述內容保護。 2020/11/27 21

2. Detailed technical description of the MPEG-4 standard 2. 1 DMIF 2. 2 Demultiplexing, buffer management and time identification 2. 3 Syntax Description 2. 4 Coding of Audio Objects 2. 5 Coding of Visual Objects 2. 6 Scene description 2. 7 User interaction 2. 8 Content-related IPR identification and protection 2. 9 Object Content Information 2020/11/27 22

Figure 3 - Major components of an MPEG-4 terminal (receiver side) 2020/11/27 23

Add figure DMIF covers three major delivery technologies 2020/11/27 24

2. 1 DMIF [con. . ] • An interactive DMIF peer as shown in Figure 4 • Is an end-system on a network which can originate a session with a target peer. • A target peer can be an interactive peer , a set of broadcast MPEG-4 streams or a set of stroed MPEG-4 files. • DMIF 提供唯一網路 session identifier 2020/11/27 26

Figure 4 The DMIF Architecture 2020/11/27 27

2. 1 DMIF [con. . ] • Through the DMIF interface can establish a multiple peer application session. • The MPEG-4 application can request from DMIF the establishment of channels with specific Qo. Ss and bandwiths for each elementary stream. • Control of DMIF spans both the Flex. Mux and the Trans. Mux layers. • 解譯和編譯成適當的網路原生信號 • 在DMIF interface 定義一般Qos參數集 2020/11/27 28

2. 2. 1 Demultiplexing • First, the channels must be located and opened. • Second, the incoming streams must be properly demultiplexed to recover the Elementary Streams from downstream channels (incoming at the receiving terminal). • The MPEG-4 demultiplexing stage is specified in terms of a conceptual two-layer multiplexer consisting of a Trans. Mux Layer and a Flex. Mux Layer as well as an Access Unit Layer that conveys synchronization information. • The Trans. Mux Layer is modeled as consisting of a protection sublayer and a multiplexing sublayer indicating that this layer is responsible for offering a specific Qo. S. 2020/11/27 29

2. 2. 1 Demultiplexing[con. . ] • Protection sublayer functionality includes error protection and error detection tools suitable for the given network or storage medium. • It provides a flexible, low overhead, low delay tool for interleaving data that may optionally be used and is especially useful when the packet size or overhead of the underlying Trans. Mux instance is large. 2020/11/27 30

Figure 5 - Buffer architecture of the System Decoder Model 2020/11/27 32

2. 4 Coding of Audio Objects 2. 4. 1 Natural Sound 2. 4. 2 Synthesized Sound 2. 4. 3 Effect 2020/11/27 33

Figure 6 - General block diagram of MPEG-4 Audio 2020/11/27 38

2. 4. 2 Synthesized Sound • Text input is converted to speech in the Text-To-Speech (TTS) decoder, while more general sounds including music may be normatively synthesized. • Synthetic music may be delivered at extremely low bitrates while still describing an exact sound signal. 2020/11/27 39

2. 4. 2 Synthesized Sound [con. . ] • It(synthetic speech) includes the following functionalities. 1. Speech synthesis using the prosody of the original speech. 2. Facial animation control with phoneme information. 3. Trick mode functionality: pause, resume, jump forward/backward. 4. International language support for text. 5. International symbol support for phonemes. 6. Support for specifying age, gender, language and dialect of the speaker. 2020/11/27 40

2. 4. 2 Synthesized Sound [con. . ] • The Structured Audio Decoder decodes input data and produces output sounds. • This decoding is driven by a special synthesis language called SAOL (Structured Audio Orchestra Language) standardized as part of MPEG-4. • This language is used to define an "orchestra" made up of "instruments" (downloaded in the bitstream, not fixed in the terminal) which create and process control data. 2020/11/27 41

2. 4. 2 Synthesized Sound [con. . ] • A score is a time-sequenced set of commands that invokes various instruments at specific times to contribute their output to an overall music performance or generation of sound effects. • The score description, downloaded in a language called SASL (Structured Audio Score Language), can be used to create new sounds, and also include additional control information for modifying existing sound. 2020/11/27 42

Add figure: The types of BIFS-Command 2020/11/27 45

The types of BIFS-Command • Define the following four basic commands: 1. Insertion 2. Deletion 3. Replacement 4. Replacement of an entire scene 2020/11/27 46

2. 5 Coding of Visual Objects 2. 5. 1 Natural Textures, Images and Video 2. 5. 2 Synthetic Objects 2. 5. 3 Structure of the tools for representing natural video 2. 5. 4 Support for Conventional and Content-Based Functionalities 2. 5. 5 The MPEG-4 Video Image and Coding Scheme 2. 5. 6 Coding of Textures and Still Images 2. 5. 7 Scalable Coding of Video Objects 2. 5. 8 Robustness in Error Prone Environments 2020/11/27 47

2. 5. 1 Natural Textures , Images and Video • the MPEG-4 standard provides solutions in the form of tools and algorithms for: 1. efficient compression of images and video 2. efficient compression of textures for texture mapping on 2 D and 3 D meshes 3. efficient compression of implicit 2 D meshes 4. efficient compression of time-varying geometry streams that animate meshes 5. efficient random access to all types of visual objects 6. extended manipulation functionality for images and video sequences 7. content-based coding of images and video 8. content-based scalability of textures, images and video 9. spatial, temporal and quality scalability 2020/11/27 48 10. error robustness and resilience in error prone environments

2. 5. 2 Synthetic Objects an initial focus the following visual synthetic objects will be described: • Parametric descriptions of a) a synthetic description of human face and body b) animation streams of the face and body • Static and Dynamic Mesh Coding with texture mapping • Texture Coding for View Dependent applications 2020/11/27 49

2. 5. 2. 1 facial animation • Facial Animation Parameter(FAPs) 是根據最少的臉部動作 及相關的臉部肌肉動作作為基礎而設計的。 • 所有的參數值中，其中包含的直線的位移(translational movement) 而這是利用 facial animation parameter units (FAPUs) 來表示的。 • Facial Definition Parameter (FDPs) 則是用來構建一個特別 的臉部模型，也就是個人化的臉部模型。 2020/11/27 50

2. 5. 2. 1 facial animation [con. . ] • 在facial definition parameter set 中包含了下列項目: 1. 3 D feature points: 臉上具有特徵的點。 2. Texture coordinates for feature points: texture coordinates 是用來配合上述的特徵點。 3. Face scene graph: 是一個臉部 3 D多邊形(polygon)的模型。 4. Face animation table: 則包含了臉部如何利用在 face scene graph 中特定的頂點之移動和FAP配合來做出臉部 表情。 2020/11/27 51

2. 5. 2. 2 body animation • 控制身體物件的資料可以分為兩種，第一種稱為BIFS ， 包含了身體定義參數 (BDP ，Body Definition Parameters); 第二種是 FBA(Facial and Body Animation) ，包含了身體 動作參數(BAPs ，Body Animation Parameters)。 • 身體定義參數包括了下列四項屬性: 1. 身體表面模型 (Body surface geometry) 2. 關節中心位置 (Joint center locations) 3. 身體表面材質貼圖 (Texture images) 4. 動作的改變 2020/11/27 52

2. 5. 2. 2 body animation [con. . ] • 身體動作參數 (Body Animation Parameter) 是用來改變身 體物件的姿勢。 2020/11/27 53

2. 5. 2. 3 2 D animated meshes • Figure 7 - 2 D mesh modeling of the "Akiyo" video object • the 2 D mesh representation of video objects enables the following functionalities: 1. Video Object Manipulation 2. Video Object Compression 3. Content-Based Video Indexing 2020/11/27 54

2. 5. 2. 4 Generic 3 D meshes • The toolbox will provide algorithms for: 1. efficient compression of generic meshes 2. (Level Of Detail) scalability of 3 D meshes - allows the decoder to decode a subset of the total bitstream to reconstruct a simplified version of the mesh containing less vertices than the original. Such simplified representations are useful to reduce the rendering time of objects which are distant from the viewer (LOD management), and also allow less powerful rendering engines to render the object at a reduced quality. 3. Spatial scalability - allows the decoder to decode a subset of the total bit stream generated by the encoder to reconstruct the mesh at a reduced spatial resolution. This feature is most useful when combined with LOD scalability. 2020/11/27 55

2. 5. 3 Structure of the tools for representing natural video • A basic classification of the bit rates and functionalities currently provided by the MPEG-4 Visual standard for natural images and video is depicted in Figure 8 below, with the attempt to cluster bit-rate levels versus sets of functionalities. 2020/11/27 56

Figure 8 - Classification of the MPEG-4 Image and Video Coding Algorithms and Tools 2020/11/27 57

2. 5. 4 Support for Conventional and Content-Based Functionalities • The MPEG-4 Video standard will support the decoding of conventional rectangular images and video as well as the decoding of images and video of arbitrary shape. This concept is illustrated in Figure 9 below. • For the content-based functionalities, where the image sequence input may be of arbitrary shape and location, this approach is extended by also coding shape and transparency information. • Shape may be either represented by an 8 bit transparency component - which allows the description of transparency if one VO is composed with other objects - or by a binary mask. 2020/11/27 58

Figure 9 - the VLBV Core and the Generic MPEG-4 Coder 2020/11/27 59

2. 5. 5 The MPEG-4 Video Image and Coding Scheme • Figure 10 below outlines the basic approach of the MPEG-4 video algorithms to encode rectangular as well as arbitrarily shaped input image sequences. • The basic coding structure involves shape coding (for arbitrarily shaped VOs) and motion compensation as well as DCT-based texture coding (using standard 8 x 8 DCT or shape adaptive DCT). 2020/11/27 60

Figure 10 - Basic block diagram of MPEG-4 Video Coder 2020/11/27 61

2. 5. 6 Coding of Textures and Still Images • Efficient Coding of visual textures and still images is supported by the visual texture mode of the MPEG-4. • This mode is based on a zerotree wavelet algorithm that provides very high coding efficiency at very wide range of bitrates. 2020/11/27 62

2. 5. 7 Scalable Coding of Video Objects • Scalability refers to the ability to only decode a part of a bit stream and reconstruct images or image sequences with: 1. reduced decoder complexity and thus reduced quality 2. reduced spatial resolution 3. reduced temporal resolution 4. with equal temporal and spatial resolution but with reduced quality. 2020/11/27 63

2. 5. 8 Robustness in Error Prone Environments 2. 5. 8. 1 Resynchronization 2. 5. 8. 2 Data Recovery 2. 5. 8. 3 Error Concealment 2020/11/27 64

2. 5. 8. 2 Data Recovery • Reversible Variable Length Codes(RVLC) • The variable length codewords are designed such that can be read both in the forward as well as the reverse direction • The parameters , QP and HEC 2020/11/27 67

Figure 11 - example of Reversible Variable Length Code 2020/11/27 68

2. 5. 8. 3 Error Concealment • Concealment strategy is highly dependent on the performance of the resychronization scheme. • For low bitrate , low delay applications the current resynchromization scheme provides very acceptable results with a simple concealment strategy , such as copying blocks from the previous frame • This approach utilizes data partitioning by separating the motion and the texture. • Requires that a second resynchroniation marker be inserted between motion and texture information. 2020/11/27 69

2. 5. 8. 3 Error Concealment • If the texture information is lost , this approach utilizes the motion information to conceal these errors. • That is , due to the errors the texture information is discarded , while the motion is used to motion compensate the previous decoded VOP. 2020/11/27 70

Figure 12: Logical structure of a scene 2020/11/27 72

2. 6 Scene description • 1. Example: How objects are grouped together: hierarchical not necessarily static node attributes can be changed while nodes can be added , replaced , or removed 2. How objects are positioned in space and time: local coordinate system fixed spatio-temporal location and scale local coordinate system 2020/11/27 73

2. 6 Scene description [con. . ] 3. Attribute Value Selection: pitch of a sound , the color for a synthetic object 4. Other transforms on AVOs: graphics primitive 2020/11/27 74

2. 7 User interaction • Client-side interaction changing the position of an object , making it visible or ivisible • Server-side interaction occur at eh transmitting end require that a back-channel is available 2020/11/27 75

2. 8 Content-related IPR identification and protection • Intellectural Property Identification (IPR) data set , carrying information about the contents , type of content and (pointers to) rights holders. • The provision of the data sets allows the implementation of mechanisms for audit trail , monitoring , billing , and copy protection. 2020/11/27 76

2. 9 Object Content Information • Allow attaching information to objects about their content. • Users of the standard can use this “OCI” datastream to send textual information along with MPEG-4 content. • It is also possible to classify content according to pre-defined tables , which will be defines outside of MPEG. 2020/11/27 77