Corso di Reti di Calcolatori II A case

Corso di Reti di Calcolatori II A case study: IPTV SLA Monitoring Giorgio Ventre The COMICS Research Group @ The University of Napoli Federico II, COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 1

Outline Ø The general problem: SLA, who cares? Ø A business case for Qo. S Ø Defining Service Level Agreements Ø A Real-Life SLA monitoring service Ø A case study: IPTV SLA Monitoring COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 2

Recent trends in the industry Ø New emerging multimedia services both in fixed and wireless networks Ø Traditional voice carriers are moving to NGN: ü Essential to control costs and drive up revenues ü Triple play services: Voice – Video – Data ü Video represents a key element of the service portfolio • Price/quality balance must attract/retain users • TV quality must compete with satellite and cable COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 3

Challenges and quality issues Ø Users are conditioned to expect high quality TV pictures: ü Users unlikely tolerate poor/fair quality pictures in IPTV ü Early delivery of broadband services is unfeasible due to the limited bandwidth compared to cable and satellite ü Compulsory data compression can potentially degrade quality Ø Need for robust transmission to minimize dataloss and delay COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 4

Why Quality Assurance is a major issue? Ø Because otherwise we wouldn’t be here Ø Quality Assurance adds a new perspective to the flatness of the current market of triple-play services Ø Quality measurement for service assurance ü End-to-end quality monitoring ü SLA based on quality delivered to end-user ü New business models and scenarios COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 5

Qo. S vs Qo. E Ø Quality of Service (Qo. S) refers to the capability of a network to provide better service to selected network traffic over various technologies. Qo. S is a measure of performance at the packet level from the network perspective. Ø Quality of Experience (Qo. E) describes the performance of a device, system, service, or application (or any combination thereof) from the user’s point of view. Qo. E is a measure of end-toend performance at the service level from the user perspective. COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 6

From Qo. S to MOS Ø MOS: Mean Opinion Score Ø Used in POTS to have a quantitative value for a “qualitative” evaluation: ü How do you evaluate the quality you perceived during your last service usage/access? Ø Very easy for simple services: telephony Ø Very complex for complex services: multimedia (sound vs video vs data vs mix) Ø Even more complex when quality of service depends on the distribution network AND terminals AND servers COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 7

Qo. S evaluation COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 8

Requirements Ø Identify parameters contributing to a satisfactory Qo. E Ø Define network performance requirements to achieve target Qo. E Ø Design measurement methods to verify Qo. E COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 9

Performance parameters Ø IPTV service is highly sensitive to packet loss Ø The impact of packet loss depends on several factors: ü Compression algorithm (MPEG 2, H. 264) ü GOP structure ü Type of information lost (I, P, B frame) ü Codec performance (coding, decoding) ü Complexity of the video content ü Error concealment at STB COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 10

Quality Measurement Ø Quality Measurement ü Objective • Pure computational • Network performance ü Objective perceptual • Measurements representative of human perception Ø Traditional metrics such as PSNR, PLR, BER are inadequate Ø Requirements for objective perceptual metrics COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 11

Why Quality-Monitoring is hard? Ø Measures have to be: ü Time-based ü Remoted ü Distributed ü Sharp Ø Highly etherogeneous environments (codecs, CPEs, media-types, …) Ø Sampled measures? ü SLAs are not sampled. Ø In order to ensure quality, measures have to be carried out with quality COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 12

Why Quality-Monitoring is hard? Ø High impact also of content based factors: Ø MPEG performance depends on content “pattern” and scene changes Ø Highly variable (movements, colours, lights) scenes generates more data Stallone vs Bergman or better Rambo vs The Seventh Seal COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 13

Methods: state of the art ØFull-Reference ØReduced-Reference ØNo-Reference COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 14

Full-reference Ø Measures are performed at both the input to the encoder and the output of the decoder Ø Both the source and the processed video sequences are available Ø Requires a reliable communication channel in order to collect measurement data COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 15

Reduced-Reference Ø Extracts only a (meaningful) sub-set of features from both the source video and the received video Ø A perceptual objective assessment of the video quality is made Ø The transmitter needs to send extracted features in addition to video data COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 16

No-Reference Ø Perceptual video quality evaluation is made based solely on the processed video sequence Ø There is no need for the source sequence Ø Measurements results are intrinsically based on a predictive model COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 17

Standards for voice quality assessment Ø ITU-T P. 862 (Feb. 2001): ü Full-reference perceptual model (PESQ) ü Signal-based measurement ü Narrow-band telephony and speech codecs ü P. 862. 1 provides output mapping for prediction on MOS scale Ø ITU-T P. 563 (May 2004): ü No-reference perceptual model ü Signal-based measurement ü Narrow-band telephony applications COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 18

Standards for voice quality assessment Ø ITU-T P. 862. 2 (Nov. 2005): ü Extension of ITU-T P. 862 ü Wide-band telephony and speech codecs (5 ~ 7 Khz) Ø ITU-T P. VQT (ongoing) ü Targeted at Vo. IP applications ü Uses P. 862 as a reference measurement ü Models analyze packet statistics; speech payload is assumed COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 19

Standards for video quality assessment Ø ITU-T J. 144 and ITU-R BT. 1683 (2004) ü Full reference perceptual model ü Digital TV ü Rec. 601 image resolution (PAL/NTSC) ü Bit rates: 768 kbps ~ 5 Mbps ü Compression errors COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 20

Standards for video quality assessment Ø IETF RFC 4445 (April 2006): A proposed Media Delivery Index (MDI) Ø MDI can be used as a quality indicator for monitoring a network intended to deliver applications such as streaming media, MPEG video, Voice over IP, or other information sensitive to arrival time and packet loss. Ø It provides an indication of traffic jitter, a measure of deviation from nominal flow rates, and a data loss at-a-glance measure for a particular video flow. COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 21

Our research Ø Objectives: ü Real-time computation of achieved quality level ü “Quality” as perceived by the user ü Per-single-user measurements ü Light computation (about +5% overhead) Ø Approach: ü Media playout and measures are both part of an integrated process ü Measurement subsystems exposes a consistent abstract interface ü Measurements results are high-level quality indicators COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 22

VQM (1/2) Ø No-Reference Ø Evaluates the video quality as perceived by the user ü Qo. S Qo. E Ø Based on MPEG 2 Ø Light parsing ü Doesn’t parse motion vectors, DCT coefficients, and other macroblock-specific information ü degradation due to packet losses is estimated using only the high-level information contained in Group of Pictures, frame, and slice headers COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 23

VQM (2/2) Ø Does not need to make assumptions concerning how the decoder deals with corrupted information ü i. e. what kind of error concealment strategy it uses. Ø Based on this information it determines exactly which slices are lost ü Go. P loss-rate ü Frame loss-rate ü Slice loss-rate ü Differentiation per frame type (I, P, B) Ø It computes how the error from missing slices propagates spatially and temporally into other slices Ø Appropriate for measuring video quality in a real-time fashion within a network COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 24

Parsing method (1/2) GOP I B B X P B B X Frame COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 25

Parsing method (2/2) MPEG-2 video bitstream 001100101101011101000010101 DECODER Quality Measurement HEADERS Decoded video stream COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II RENDERING 26

Qo. E vs. MOS Ø Mapping between Quality of Experience evaluation and MOS (Mean Opinion Score – ITU/T P. 800) value MOS 5 4 3 2 1 QMAX Qo. E COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 27

MOS vs SLAs Ø Knowledge of the function MOS(t) directly enables SLAs monitoring DOWN TIME 5 4 MOS 3 2 1 SLA TRESHOLD TIME COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 28

Experimental testbed Controlled-Loss Router Video Dropped Video Client Server Packets + Quality Meter Video Characteristics: MPEG 2 -TS Constant Bit Rate: 3. 9 Mbps COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 29

High Quality Throughput: 5. 0 Mbps COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 30

Medium Quality Throughput: 3. 9 Mbps COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 31

Low Quality Throughput: 3. 0 Mbps COMICS (COMputer for Interaction and Communication. S) Research Group – DIS, University of Napoli Federico II 32