CS 598 KN Advanced Multimedia Systems Multimedia Protocols

CS 598 KN – Advanced Multimedia Systems Multimedia Protocols Klara Nahrstedt September 17, 2018 CS 598 KN - Fall 2018

Internet Multimedia Protocol Stack APPLICATION Media encapsulation (H. 264, H. 265, HVEC, MPEG-4) DASH Web. RTC SIP RTSP RSVP RTCP RTMP QUIC RTP HTTP KERNEL TCP DCCP UDP IP Version 4, IP Version 6 AAL 3/4 AAL 5 MPLS ATM/Fiber Optics Layer 5 (Session) Layer 4 (Transport) Layer 3 (Network) Layer 2 (Link/MAC) Ethernet/Wi. Fi/Bluetooth/ Zigbee/… CS 598 KN - Fall 2018

Service Requirements for Realtime Flows (Voice/Video) Sequencing n Intra-media synchronization n Inter-media synchronization n Payload identification n Frame indication n CS 598 KN - Fall 2018

MULTIMEDIA TRANSPORT PROTOCOLS CS 598 KN - Fall 2018

Real-time Transmission Protocol (RTP) n n RTP provides end-to-end transport functions suitable for real-time audio/video applications over multicast and unicast network services RTP companion protocol – Real-time Transport Control Protocol (RTCP) RTP RTCP User Datagram Protocol Internet Protocol Ethernet 802. 13 or Wi-Fi 802. 11 PHY (Wired or Wireless) CS 598 KN - Fall 2018 Layer 4

Relation between RTP and RTCP Application Decoding Coding RTP RTCP Application Coding RTCP UDP/IP CS 598 KN - Fall 2018 Decoding RTP

RTCP: Control and Management n Out-of-band control information for RTP flow. ¨ Monitors Qo. S for RTP in the delivery and packaging of multimedia data ¨ Used periodically to transmit control packets to participants in a streaming multimedia session. ¨ Provides feedback on the quality of service being provided by RTP. ¨ Gathers statistics on media connection n n Bytes sent, packets sent, lost packets, jitter, feedback and round trip delay. Application may use this information to increase the quality of service, perhaps by limiting flow or using a different codec. CS 598 KN - Fall 2018

RTCP Functions n There are several type of RTCP packets: ¨ Sender report packet, ¨ Receiver report packet, ¨ Source Description RTCP Packet, ¨ Goodbye RTCP Packet and ¨ Application Specific RTCP packets. n RTCP itself does not provide any flow encryption or authentication means. SRTCP protocol can be used for that purpose. CS 598 KN - Fall 2018

RTP Services n Payload Type Identification ¨ Determination of media coding ¨ Source identification ¨ RTP works with Profiles n n Profile defines a set of payload type codes and their mappings to payload formats Sequence numbering ¨ Error n Time-stamping ¨ Time n detection monitoring, synchronization, jitter calculation Delivery monitoring CS 598 KN - Fall 2018

RTP Services – Support of Heterogeneity n Mixer service ¨ Allows for resynchronization of incoming audio packets ¨ Reconstructs constant 20 ms spacing generated by sender ¨ Mixes reconstructed audio streams into single stream ¨ Translates audio encoding to lower bandwidth ¨ Forwards lower bandwidth packet streams n Translator service ¨ Allows for translation between IP and other high speed protocols ¨ May change encoding data CS 598 KN - Fall 2018

Applications of RTCP/RTP n n RTP usage – in several application audio and video tools (vat, vic) RTP follows the principle of application level framing and integrated layer processing RTP/UDP/IP is being used by the current streaming session protocols such as RTSP Multimedia applications such as Quick. Time, Real Player and others use them CS 598 KN - Fall 2018

SESSION CONTROL PROTOCOLS FOR VIDEO STREAMING: RTSP, (SDP, SAP) CS 598 KN - Fall 2018

Real-Time Streaming Protocol (RTSP) n n Application Protocol for Control of multimedia streams This is not an application data transmission protocol, just remote control protocol between client and server Audio Video Decoder Session Control RTSP RTP Audio video Coder RTP CLIENT SERVER CS 598 KN - Fall 2018

RTSP n n Approved as Internet Draft, February 2, 1998, authors H. Schulzrinne, A. Rao, R. Lanphier Enables controlled, on-demand delivery of realtime data such as audio and video Intends to control multiple data delivery sessions Provides means for choosing delivery channels ¨ UDP ¨ Multicast UDP, ¨ TCP CS 598 KN - Fall 2018

RTSP Methods Request Direction Description OPTIONS S <-> C Determine capabilities of server (S) or client (C) DESCRIBE C -> S Get description of media stream ANNOUNCE S <-> C Announce new session description SETUP C -> S Create media session RECORD C -> S Start media recording PLAY C -> S Start media delivery PAUSE C -> S Pause media delivery REDIRECT S -> C Use other server TEARDOWN C -> S Destroy media session SET_PARAMETER S <-> C Set server or client parameter GET_PARAMETER S <-> C Read server or client parameter CS 598 KN - Fall 2018

RTSP – Aggregate and Stream Control DESCRIBE (aggregate stream) RTSP Decoder SETUP RTSP SID PLAY RTP Coder RTP ESTABLISH CLIENT PLAY CS 598 KN - Fall 2018 SERVER

Streaming media: meta files client CS 598 KN - Fall 2018 RTSP Presentation by H. Schulzrinne, 2001

Streaming Media: RTSP CS 598 KN - Fall 2018 RTSP Presentation by H. Schulzrinne, 2001

RTSP Operation CS 598 KN - Fall 2018 RTSP Presentation by H. Schulzrinne, 2001

VOICE OVER IP PROTOCOLS CS 598 KN - Fall 2018

Internet Telephony France (1970) Videophony – imagined in 1910 AT&T Picture-Phone in 1969 Source: http: //en. wikipedia. org/wiki/Videophone Avaya IP Phone CS 598 KN - Fall 2018

Signaling for IP Telephony n n Internet Telephone – needs ability of one party to signal to other party to initiate a new call Call – association between a number of participants ¨ Note: there is no physical channel or network resources associated with the session layer connection, the connection exists only as signaling state at two end points CS 598 KN - Fall 2018

IP Telephony Signaling Protocol (Requirements) n Name translations and user location ¨ Mapping between names of different levels of abstraction ¨ n Email address to IP address of host Feature negotiation ¨ Group of end systems must agree on what media to exchange ad their respective parameters ¨ n Different encodings, rates Call Participant Management n Invite participants to existing call, transfer call and hold other users CS 598 KN - Fall 2018

IP Telephony Signaling (Requirements) n Feature change ¨ Adjust composition of media sessions during the course of call Add or reduce functionality n Impose or remove constraints due to addition or removal of participants n n Two signaling protocols: ¨ SIP (IETF Standard) ¨ H. 323 (ITU Standard) CS 598 KN - Fall 2018

SIP (Session Initiation Protocol) n n n SIP Goal: invite new participants to call Client-Server protocol at the application level Protocol: User/Client creates requests and sends to server; ¨ User agent server responds; ¨ n n SIP requests can traverse many proxy servers Server may act as redirect server Proxies or redirect servers cannot accept/reject requests, only user agent server can Requests/Responses are textual CS 598 KN - Fall 2018

Call Setup Process using SIP Location Service (3) Where is johnsmith? (1) INVITE sip: johnsmith@test. com (4) At Jsmith (2) INVITE sip: johnsmith@test. com (8) 200 OK (5) INVITE sip: johnsmith@jsmith. test. com (6) 200 OK (7) 200 OK SIP Server (9) ACK Jsmith SIP user agent (10) RTP Audio/Video data CS 598 KN - Fall 2018 SIP user agent

SIP Redirect Server Operation Location Service (2) Where is johnsmith? (3) At play (1) INVITE sip: johnsmith@test. com (4) 302 Moved temporarily Contact: sip: johnsmith@play. test. com (5) INVITE sip: johnsmith@play. test. com (6) 200 OK SIP user agent (7) RTP Audio/Video data CS 598 KN - Fall 2018 play SIP user agent

SIP - Message Calls in SIP – have unique call ID (carried in Call-ID header field of SIP message) n Call identifier is created by the caller and used by all participants n SIP messages have information n ¨ Logical connection source ¨ Logical connection destination ¨ Media capabilities (use SDP) CS 598 KN - Fall 2018

SIP – Addressing and Naming n n To be invited and identified, called party must be named SIP chooses email-like identifier user@domain ¨ user@host ¨ user@IPaddress ¨ phone-number@gateway ¨ n SIP’s address: part of SIP URL sip: j. doe@example. com ¨ URL can be placed on web page ¨ n Interactive audio/video requests translation ¨ name@domain to host@host CS 598 KN - Fall 2018

SIP Requests/Methods n n n INVITE—Indicates a client is being invited to participate in a call session. ACK—Confirms that the client has received a final response to an INVITE request. BYE—Terminates a call and can be sent by either the caller or the callee. CANCEL—Cancels any pending searches but does not terminate a call that has already been accepted. OPTIONS—Queries the capabilities of servers. REGISTER—Registers the address listed in the To header field with a SIP server. CS 598 KN - Fall 2018

SIP Responses n 1 xx—Informational Responses ¨ ¨ ¨ n 100 Trying (extended search being performed may take a significant time so a forking proxy must send a 100 Trying response) 180 Ringing 181 Call Is Being Forwarded 182 Queued 183 Session Progress 2 xx—Successful Responses 200 OK ¨ 202 accepted: It Indicates that the request has been understood but actually can't be processed ¨ n 3 xx—Redirection Responses 300 Multiple Choices ¨ 301 Moved Permanently ¨ 302 Moved Temporarily ¨ CS 598 KN - Fall 2018

MULTIMEDIA STREAMING PROTOCOL - DASH (DYNAMIC ADAPTIVE STREAMING HTTP) CS 598 KN - Fall 2018

Reason for DASH: Mobile Video Streaming Challenges Mobile Internet use is expanding dramatically n Video traffic is growing exponentially n Challenges: n ¨ Mobile users expect high quality video experience ¨ Network operators need to offer quality experience affordably CS 598 KN - Fall 2018

Growth of Mobile Internet/Video Thomas Stockhammer, Qualcomm, “DASH – Design Principles and Standards , MMSys 2011 CS 598 KN - Fall 2018

Need: Adaptive Streaming Concept n Adaptive Streaming technologies enable ¨ Optimal streaming video viewing experience for diverse range of devices over broad set of connection speeds n Adaptive streaming technologies share ¨ Production of multiple files from the same source file to distribute to viewers watching on different powered devices via different connection speeds ¨ Distribution of files adaptively, changing stream that is delivered to adapt to changes in effective throughput and available CPU cycles on playback stations ¨ Transparent operation to the user so that the viewer clicks one button and all streams switch/adapt behind the scenes. CS 598 KN - Fall 2018

Adaptive Streaming One Approach of Adaptive Streaming 1. Server sends first the high important video information (e. g. , I frames) And after the high importance video information is sent, lower importance video information follows (e. g. , P and B frames) if bandwidth and time allows Second Approach of Adaptive Streaming 1. Server sends with high quality part of the frame and only progressively , if bandwidth and time allow, it sends the rest of the frame information Third Approach of Adaptive Streaming 1. At server video is encoded in multiple bitrates and depending on the device Bandwidth it adjusts at what rate Other approaches exist CS 598 KN - Fall 2018 Source: http: //www. dicomdistribution. com/Adaptive%20 Streaming. html

Standardization History Thomas Stockhammer, Qualcomm, “DASH – Design Principles and Standards , CS 598 KN - Fall 2018 Presentation at MMSys 2011

Adaptive HTTP Streaming System (Protocol) n Server ¨ Can n be standard web server ¨ Media segment can be prepared in-line or offline Client Sends series of HTTP GET segment requests and receives segments ¨ Performs rate adaptation before sending a new GET segment request ¨ CS 598 KN - Fall 2018

Client-centric approach Client has best view of network conditions n No session state in network n ¨ Redundancy ¨ Scalability Faster innovation and experimentation n But, relies on client for operational metrics n ¨ Only client knows what really happens CS 598 KN - Fall 2018

Terms and Definitions of Adaptive HTTP Streaming n Need ¨ Media Presentation Description (MDP) which provides metadata For requesting (GET request) media segments n For rate adaptation purpose n ¨ Segment which may include media data or metadata to decode n Need DASH CS 598 KN - Fall 2018

DASH – Dynamic Adaptive Streaming over HTTP n Dash is NOT ¨ System, n protocol, presentation, codec, interactivity What is DASH ¨ ¨ ¨ Enabler which provides formats to enable efficient and highquality delivery of streaming services over the Internet Component of end-to-end service Enabler to reuse existing technologies (containers, DRM (Digital Rights Management), codecs) Enabler for deployment on top of HTTP-CDNs Enabler for very high user experience (low start-up, no rebuffering) Provides simple inter-operability points (profiles) CS 598 KN - Fall 2018

DASH Client Thomas Stockhammer, Qualcomm, “DASH – Design Principles and Standards , CS 598 KN - Fall 2018 Presentation at MMSys 2011

Information Classification n n DASH uses MPD (Media Presentation Descriptor) and Index Information as metadata for DASH Access Client Initialization and Media Segments for Media Engine ¨ Reuse of existing container format Source: MMSys’ 11 CS 598 KN - Fall 2018

Media Presentation Data Model MDP - description of accessible segments and corresponding timing CS 598 KN - Fall 2018

MDP Information Includes redundant information of media streams to initially select or reject groups or representations n Includes access and timing information n ¨ Content addressing via HTTP-URLs ¨ Byte range for each accessible segment ¨ Segment availability start and end time in wall-clock time ¨ Approximate media start time and duration ¨ Instructions on starting playout (for live service) n Includes switching relations across representations CS 598 KN - Fall 2018

Media Segments (1) Contain information to map segment into media presentation timeline for switching and synchronous presentation with other representations n Can be short (~ 1 -10 seconds) n Can be long (~10 sec – 2 hours) n CS 598 KN - Fall 2018

Media Segments (2) Media segment duration advantages Short duration Commonality with live - Large number of files high switching granularity - Large number of on segment level URLs - Fixed request size - Switching granularity on segment level Long duration - Small number of files - Small number of URLs -High switching granularity -Flexible request sizes -Improved cache performance CS 598 KN - Fall 2018 disadvantages - Need for segment index - Difference from live

Segment Indexing n Provides information in ISO box structure on ¨ Accessible units of data (e. g. , frames) in media segment ¨ Byte range in segments (easy access through HTTP GET) ¨ Accurate presentation duration (seamless switching) ¨ Presence of representation access positions n Provides compact bitrate-over-time to client ¨ Can n n be used for intelligent request schedule Generic data structure Hierarchical structuring for efficient access CS 598 KN - Fall 2018

Media Segment with Segment Index CS 598 KN - Fall 2018

DASH Capabilities n n n Enables live, on-demand time-shift services Allows independency of request sizes and segment sizes Allows segment formats that are ¨ ISO base media formats – ISO BMFF or MPEG-2 TS n n ISO BMFF -. . File Format (extensions) MPEG-2 TS -. . Transport Stream (extensions) Guidelines for integrating any other format ¨ Codec independent ¨ n n Supports server/client component synchronization (e. g. , separate and multiplexed AV) Enables targeted ad insertion Provides definition of quality metrics Enables content descriptors for protection, accessibility, rating, … CS 598 KN - Fall 2018

Conclusion Extensively usage of various multimedia protocols n Metadata used by DASH and session mobility approach use MPEG-21 international standard n ¨ Facilitates MPEG-21 Digital Item ¨ Leverages existing media repositories such as UPn. P n One possible implementation of DASH ¨ http: //www-itec. uni-klu. ac. at/dash CS 598 KN - Fall 2018