Internetbased interactive HDTV Bin Yu Klara Nahrstedt Department

Internet-based interactive HDTV Bin Yu, Klara Nahrstedt Department of Computer Science University of Illinois at Urbana-Champaign ACM/Springer Multimedia Systems Journal, 9(5) March, 2004 9/29/2020 DING Feng HT 040297 J 1

Interactive HDTV – Traditional Approach l TV cable network + set-top box + TV set ¡ High quality video delivery ¡ Not compatible, proprietary format ¡ Not scalable, all processing is done in the TV studio ¡ Interactivity, flexibility and customization are not easy 9/29/2020 DING Feng HT 040297 J 2

Interactive HDTV – Traditional Approach 9/29/2020 DING Feng HT 040297 J 3

Interactive HDTV – Traditional Approach l PC + Internet ¡ Easy interactivity, flexibility and customization ¡ No high quality HDTV display ¡ Current software solutions are inefficient 9/29/2020 DING Feng HT 040297 J 4

HDControl – Authors’ Approach l Internet + PC + TV set ¡ Combine good features of previous solutions ¡ PC function as open set-top box ¡ Standard video format: MPEG 2 ¡ Distributed architecture 9/29/2020 DING Feng HT 040297 J 5

HDControl – Architecture 9/29/2020 DING Feng HT 040297 J 6

Typical Application l Picture-in Picture (Pi. P) l Logo/ticker insertion l Email browsing l… 9/29/2020 DING Feng HT 040297 J 7

Visual Information Embedding (VIE) l Naïve solution – not feasible ¡ Decode the video stream into raw pixels ¡ Overlay embedded content ¡ Re-encode to MPEG 2 stream 9/29/2020 DING Feng HT 040297 J 8

MPEG 2 Encoder Diagram – Recall 9/29/2020 DING Feng HT 040297 J 9

Typical GOP Structure 9/29/2020 DING Feng HT 040297 J 10

Visual Information Embedding l l l Inverse Variable Length Coding (VLC) converts bit-stream to Motion Compensation (MC) domain MC contains motion vectors and prediction errors (residual) in quantized DCT format Do MC to get reconstructed DCT (RD) domain data Replace part of the image with embedded information Re-encode data to MPEG 2 format 9/29/2020 DING Feng HT 040297 J 11

VIE l Wrong ¡ MB 2 9/29/2020 reference problem use MB 1 as a reference for prediction DING Feng HT 040297 J 12

VIE l Fix reference ¡ RD domain data of MB 2, MB 1 and MB 0 are required ¡ Worst case: IBBPBBPBB ¡ Maximum search distance: 16 macroblocks ¡ Potentially all the MBs in I and P frames need to be decoded for future use 9/29/2020 DING Feng HT 040297 J 13

Two Types of Macroblocks l d-MBs ¡ MB 0 and MB 1, their data should be decoded to RD domain for future reference l c-MBs ¡ MB 2, their reference blocks are wrong and their MC data have been changed 9/29/2020 DING Feng HT 040297 J 14

Efficient Motion Compensation l Only those macroblocks surrounding the foreground window are affected by VIE (Chang et al. ) l Future motion prediction pattern is unknown l Completely reconstruct the reference frame 9/29/2020 DING Feng HT 040297 J 15

Efficient Motion Compensation l Introduce delay l Buffering a GOP l All c-MBs can be detected by testing whether their reference MBs are in the foreground area l The reference MBs for both c-MBs and d. MBs are d-MBs 9/29/2020 DING Feng HT 040297 J 16

Efficient Motion Compensation l Back tracking ¡ c-MB -> d-MB -> … ->d-MB ¡ Perform motion compensation for c-MBs and d. MBs to get RD domain data ¡ Perform motion estimation for c-MBs to get their new motion vectors and prediction errors 9/29/2020 DING Feng HT 040297 J 17

Optimizations l Bi-direction prediction -> uni-direction prediction ¡B frames has two reference frames ¡ One reference frame is in the foreground area ¡ Delete one motion vector 9/29/2020 DING Feng HT 040297 J 18

Optimizations l Mark out sensitive area ¡ Foreground window may only occupy a small area ¡ Define c-sensitive area which may contains c. MBs; define d-sensitive are which may contains d-MBs. ¡ For slices in insensitive area, copy them directly from input to output without decoding 9/29/2020 DING Feng HT 040297 J 19

Optimizations l Shorten the delay ¡ ¡ Select a shorter GOP size Start back tracking process earlier l l 9/29/2020 The sensitive area for the last P frame is mainly within the foreground window IBBPBBPBB DING Feng HT 040297 J 20

Resynchronization l MPEG 2 ¡ Decoding timestamp (DTS) and presentation timestamp (PTS) for each frame are determined by sender clock ¡ Synchronization point (PCR) in multiplexed stream ¡ Distance between PCR is constant 9/29/2020 DING Feng HT 040297 J 21

De-multiplexing and Multiplexing 9/29/2020 DING Feng HT 040297 J 22

Resynchronization l VIE problem ¡ The frame size will be changed by VIE process ¡ The constant spacing of PCR will be violated 9/29/2020 DING Feng HT 040297 J 23

Two solutions l Simple padding ¡ Pad Null packet in the empty space to keep the PCR positions ¡ Can only handle bit-rate reduction ¡ Waste bandwidth l Time-invariant bit-rate scaling ¡ To scale the bit-rate to another constant value with scaling factor SF 9/29/2020 DING Feng HT 040297 J 24

Determine SF Measure the maximal ratio (R) in number of packets before and after VIE offline l Monitor the actual ratio (AR) online l New SF l 9/29/2020 DING Feng HT 040297 J 25

Evaluation l Experimental ¡ HDTV environment testbed Live high-definition digital TV stream from the satellite or storage device is fed into the server l Encode video into MPEG 2 format and then multicast over LAN l Player PC decode the stream and send it to TV set l Video editing servants receive the stream and perform editing operation l Multicast result stream l 9/29/2020 DING Feng HT 040297 J 26

Pi. P FG: football_sd. mpg(480*256, 30 fps) BG: trees 1. mpg(1920*1088, 30 fps) 9/29/2020 DING Feng HT 040297 J 27

Evaluation l Distribution 9/29/2020 of d-MBs DING Feng HT 040297 J 28

Comparison with the previous approach The number of MBs to be convert from MC domain to RD domain 9/29/2020 DING Feng HT 040297 J 29

Resynchronization results 9/29/2020 DING Feng HT 040297 J 30

Thank you! 9/29/2020 DING Feng HT 040297 J 31