Streaming Video Over the Internet Andreas Panteli 03052012

Streaming Video Over the Internet Andreas Panteli 03/05/2012 ECE 654 Advanced Computer Networks 1 Streaming Video Over The Internet 03 May 2012

Introduction § Video principles § § § Streaming Video § § § Video content need not be downloaded in full It is being played out while parts of the content are being received and decoded Real-time multimedia have timing constraints § § § When an image appears on the retina it is retained for some millisecond before decaying If a sequence of images is drawn line by line at 50 images/sec the eye does notice that it is looking at discrete images Audio and video data must be played out continuously Bandwidth, delay and loss requirements No Qo. S guarantees to streaming video, from the current best-effort Internet! 2 Streaming Video Over The Internet 03 May 2012

An Architecture for Video Streaming Server Client/Receiver Video Decoder Storage Device Raw Video Raw Audio Video Compression Audio Compression Compressed Video Compressed Audio Application-layer Qo. S Control Transport Protocol Media Synchronization Audio Decoder Application-layer Qo. S Control Transport Protocol Internet (Continuous media distribution services) 3 Streaming Video Over The Internet 03 May 2012

Key Areas § Video Compression § Application-Layer Qo. S Control § Continuous Media Distribution Services § Media Synchronization Mechanisms § Protocols for Streaming Media 4 Streaming Video Over The Internet 03 May 2012

Key Areas § § § Video Compression § § JPEG standard MPEG standard Application-Layer Qo. S Control Continuous Media Distribution Services Media Synchronization Mechanisms Protocols for Streaming Media 5 Streaming Video Over The Internet 03 May 2012

Digital Video § § Digital video is represented as a sequent of frames, each consisting of a rectangular grid of picture elements (pixels) For color video, 8 bits for each of the RGB colors are being used § To produce smooth motion, digital video, we must display at least 25 frames/sec, but to avoid flickering we need to repaint each frame on the screen twice Smoothness and flickering affects the bandwidth requirements § § 24 bits/pixel => 16 million colors Consider the lower video configuration for computer monitor 1024 x 768 with 24 bits/pixel and 25 frames/sec § Need to be fed at 472 Mbps We need video compression! 6 Streaming Video Over The Internet 03 May 2012

Video Compression § Two algorithms are needed § § Encoding and Decoding Asymmetries of the algorithms § § 7 Acceptable for the encoding algorithm to be slow and require expensive HW, if the decoding algorithm is fast § Only for video on demand § For real-time multimedia like video conferencing, slow encoding is unacceptable Acceptable to have a slightly different video signal than the original, after encoding and decoding (Lossy system) § Lossy systems are important § Accepting a small amount of information loss can give a huge payoff in terms of the compression ratio possible Streaming Video Over The Internet 03 May 2012

The JPEG Standard § Compression algorithm for still images § § Compute the luminance Y and the two chrominances I and Q § § Y = 0. 30 R + 0. 59 G + 0. 11 B I = 0. 60 R – 0. 28 G - 0. 32 B Q = 0. 21 R – 0. 52 G + 0. 31 B Compress the two chrominances twice than the luminance § § § Could be applied to each image in succession to achieve video compression Lossy reduction, but the eye barely notices it since it responds to luminance more than to chrominance JPEG often produces 20: 1 compression or better Decoding a JPEG image requires running the algorithm backwards 8 Streaming Video Over The Internet 03 May 2012

The MPEG Standard (1/2) § § Main algorithms for video compression and international standards since 1993 Can compress both audio and video § Three parts in MPEG-1 § § § 9 Audio Video System (integrates the other two) Streaming Video Over The Internet 03 May 2012

The MPEG Standard (2/2) § Two kinds of redundancies in videos § § § MPEG-1 output consists of four kinds of frames § § § Spatial redundancy § Coding each frame separately with JPEG Temporal redundancy § Consecutive frames are often almost identical I (Intracoded) frames: Self-contained JPEG-encoded still pictures P (Predictive) frames: Block-by-Block difference with the last frame B (Bidirectional) frames: Differences between the last and the next frame D (DC-coded) frames: Block averages used for fast-forward MPEG-1 can achieve a compression factor of 40 Decoding has about the same complexity as encoding 10 Streaming Video Over The Internet 03 May 2012

Key Areas § § § Video Compression Application-Layer Qo. S Control § § Congestion control Error control Continuous Media Distribution Services Media Synchronization Mechanisms Protocols for Streaming Media 11 Streaming Video Over The Internet 03 May 2012

Application-Layer Qo. S Control § Avoids congestion and maximizes video quality in the presence to packet loss Congestion Control § Rate Control § § § 12 § Source-Based Rate Control Receiver-Based Rate Control Hybrid-Rate Control Rate Shaping § § Error Control § FEC § Delay-Constrained Retransmission § Error-Resilient Encoding § Error-Concealment Frame-dropping filter Layer-dropping filter Frequency filter Re quantization filter Streaming Video Over The Internet 03 May 2012

Rate Control (1/2) § § Technique to determine the sending rate of video traffic based on the estimated available bandwidth of the network Source-based rate control § § § 13 The sender is responsible for adapting the video transmission rate based on feedback from the receivers Probe-based approach § Sending rate adjustment to maintain the packet loss ratio below a certain threshold Model-based approach Streaming Video Over The Internet 03 May 2012

Rate Control (2/2) § Receiver-based rate control § § The receivers regulate the receiving rate of video streams by adding/dropping channels Probe-based approach § When no congestion is detected, join a channel § => increase of it’s receiving rate § When congestion is detected, drop a channel § => reduction of it’s receiving rate Model-based approach § Explicit estimation for the available network bandwidth § Also based on Equation (1) Hybrid rate Control § 14 Combination of the above mentioned techniques Streaming Video Over The Internet 03 May 2012

Rate Shaping § § § Matches the rate of a pre-compressed video bit stream to the target rate constraint A rate shaper (or filter) is required for source based rate control Types of filters § § 15 Codec filter § Performs transcoding between different compression schemes Frame-dropping filter § I, P , B , D frames in MPEG Frequency filter § Operates in the frequency domain (DCT coefficients) Re-quantization filter § Re-quantizes the DCT coefficients with a larger quantization step Streaming Video Over The Internet 03 May 2012

Error Control (1/2) § Forward Error Correction (FEC) § § 16 Add redundant information so that original message can be reconstructed in the presence of packet loss Channel coding § Stream is chopped into segments § Each segment is packetized into k packets § A block code generates an n-packet block, where n>k § User only needs to receive any k packets out of n of the block Source coding § N-th group of blocks contains redundant information of (n-1)th group of blocks Joint source/channel coding § Optimal rate allocation between Channel and Source coding Streaming Video Over The Internet 03 May 2012

Error Control (2/2) § Delay-constraint Retransmission When the receiver detects the loss of packet N if (Tc + RTT + Ds < Td(N)) send the request for packet N to the sender Where Tc current time RTT estimated round-trip time Ds a slack term (e. g receiver’s decoding delay) Td(N) time when packet N is scheduled for display § § Error-resilient Encoding § § Error Concealment § § § 17 Enhances robustness of compressed video to packet loss A video is compressed into multiple streams (descriptions) Each description provides acceptable visual quality Combined descriptions provide a better visual quality Performed by the receiver when packet loss has already occurred Spatial interpolation § Missing pixel values are reconstructed using neighboring pixels Temporal interpolation § Loss data is reconstructed using data from the previous frame Streaming Video Over The Internet 03 May 2012

Key Areas § § § Video Compression Application-Layer Qo. S Control Continuous Media Distribution Services § § § Network Filtering Application-level Multicast Content Replication Media Synchronization Mechanisms Protocols for Streaming Media 18 Streaming Video Over The Internet 03 May 2012

Network Filtering § Adequate support from the network is critical in order to provide quality multimedia presentations § Routers have no knowledge of the format of the media streams § § § R: Router Randomly discard packets Filters receive the client’s requests and adapt the stream sent by the server accordingly Typically, frame-dropping filters are used § § 19 Drop packets in a way that gracefully degrades the stream’s quality Increase bandwidth efficiency by discarding late frames Streaming Video Over The Internet 03 May 2012

Application-level Multicast Unicast § § Multicast Internet’s original design fails to effectively support streaming-media multicast IP multicast has barriers § Scalability, network management, support for higher layer functionality (e. g. error, flow and congestion control) Application-level multicast to the rescue § Builds a multicast service on top of the Internet Media bridge § § § 20 Routing in the application layer Each Media Bridge is interconnected with one or more neighboring Media. Bridges through explicit configuration Collectively employ a distributed application-level multicast routing algorthm Streaming Video Over The Internet 03 May 2012

Content Replication § Advantages § Increased network’s bandwidth efficiency § Reduced load on streaming servers § Reduced latency for clients § Increased availability Mirroring Caching § § 21 Scatter copies of the original multimedia files in different locations around the Internet Clients retrieve data from nearest duplicate server § § Exploits temporal locality Cache retrieves data from server Clients retrieve data from cache if available Cache sharing and cache hierarchies Streaming Video Over The Internet 03 May 2012

Key Areas § § § Video Compression Application-Layer Qo. S Control Continuous Media Distribution Services Media Synchronization Mechanisms § § § Intra-stream Synchronization Inter-object Synchronization Protocols for Streaming Media 22 Streaming Video Over The Internet 03 May 2012

Media Synchronization § § § Multimedia applications § Various integrated media streams that must be presented in a synchronized fashion Media streams may lose synchronization after moving from the server to the client Three layers of synchronization § § § 23 Intra-stream synchronization § Maintain continuity of logical data units Inter-stream synchronization § Maintain temporal relationships among different continuous media § e. g. movements of the lips of a speaker do not correspond to the presented audio Inter-object synchronization § Relation with time-independent data, such as text, still image and streams § e. g. during a presentation, audio is commenting one slide while another slide is presented Streaming Video Over The Internet 03 May 2012

Media Synchronization Mechanisms § § § Delay in the network is unpredictable due to the best-effort nature of the Internet Axes-based specifications, or time-stamping § At the source, a stream is time-stamped to keep temporal information within the stream and with respect to other streams Mechanisms implemented on the end systems § § 24 Preventive mechanisms § Minimize latencies and jitters § i. e. disk-reading scheduling algorithms, network transport protocols, operating systems and synchronization schedulers Corrective mechanisms § Recover synchronization in the presence of synchronization errors § i. e. Stream synchronization protocol (SSP) § Concept of ”intentional delay” Streaming Video Over The Internet 03 May 2012

Key Areas § Video Compression § Application-Layer Qo. S Control § Continuous Media Distribution Services § Media Synchronization Mechanisms § Protocols for Streaming Media § § 25 Transport Protocols Session Control Protocols Streaming Video Over The Internet 03 May 2012

Protocols for Streaming Video § Data Plane § § Compressed video/audio is packetized at the real-time transport protocol (RTP) layer RTP packetized streams are then passed to the UDP/TCP layer and IP layer IP packets are transported over the Internet Control Plane § § 26 Real-time control protocol (RTCP) and real-time streaming protocol (RTSP) packets are multiplexed at the UDP/TCP layer IP packets are transported over the Internet Streaming Video Over The Internet 03 May 2012

Transport Protocols § UDP and TCP provide basic transport functions § § § UDP and TCP § Multiplexing, Error control (e. g. checksum) TCP only: § Retransmission, Congestion control, Flow control UDP is typically employed as the transport protocol for video streams § No packet delivery guarrantees § Receiver needs to rely in upper layer to detect packet loss RTP and RTCP run on top of UDP/TCP RTP : does not guarantee Qo. S or reliable delivery § Time stamping § Sequence numbering § Payload type identification § Source identification 27 § RTCP : designed to work in conjunction with RTP § Qo. S feedback § Participant identification § Control packets scaling § Inter-media synchronization § Minimal session control information Streaming Video Over The Internet 03 May 2012

Session Control Protocols § RTSP (Real-Time Streaming Protocol) § § Support VCR-like control operations § (stop, pause/resume, fast forward, fast backward) Choosing delivery channel (UDT, multicast UDT, TCP) Establish and control streams of continuous audio and video media between the media servers and the clients § Media retrieval: ask server to setup a session to send the requested media data § Adding media to an existing session: server or client can notify each other about any additional media becoming available to the established session SIP (Session Initiation Protocol) § § 28 Similar to RTSP Support user mobility by proxying and redirecting request Streaming Video Over The Internet 03 May 2012

References § § A. S. Tanenbaum, Computer Networks, 4 th ed. New Jersey: Prentice Hall, 2002. Dapeng Wu, Hou, Y. T. , Wenwu Zhu, Ya-Qin Zhang, Peha, J. M. , "Streaming video over the Internet: approaches and directions“, Circuits and Systems for Video Technology, IEEE Transactions on , vol. 11, no. 3, pp. 282 -300, Mar 2001 Dapeng Wu, Yiwei Thoms Hou, Ya-Qin Zhang, "Transporting real-time video over the Internet: challenges and approaches, " Proceedings of the IEEE , vol. 88, no. 12, pp. 1855 -1877, Dec 2000 Dapeng Wu and Y. Thomas Hou and Jason Yao and Y. Thomas and Hou Jason Yao and H. Jonathan Chao, “Realtime Video over the Internet: A Big Picture”, IEEE Net. World+Interop, 2000 29 Streaming Video Over The Internet 03 May 2012

Questions? Thank you! 30 Streaming Video Over The Internet 03 May 2012