Quality of Service of OverTheTop Services Cyril Lau

Quality of Service of Over-The-Top Services Cyril Lau ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 1

Presentation Objective and Agenda Presentation Objective § To discuss the Quality of Experience (Qo. E) aspects of over-the-top (OTT) and social media services Agenda § Rise of OTT and mobile social media apps § Factors affecting subscriber experience and quality § Case study – OTT Qo. E measurement § Whatsapp, Facebook, youtube § Q&A ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 2

[ ] OTT SERVICES OVERVIEW ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 3

Rise of OTT services - OTT driven by: - faster, more reliable data connectivity on cellular mobile networks - Smartphone devices with better performance, interfaces and larger screens - Better codec development for voice and video transmission - Method of accessing a new market of users formerly controlled solely by operators 340 m users ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 1 bn users 4

Types of OTT Services - Social Media - Twitter, Facebook, Instagram - Communications - Viber, Skype, Whats. App - Content - Netflix, You. Tube - Trend is that social media and communications functions are consolidating (Facebook purchase of Whats. App etc) ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 5

[ ] FACTORS AFFECTING QUALITY OF EXPERIENCE ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 6

The Carrier Challenge § Service quality is critical to maintaining subscriber loyalty, even though some areas are outside operator’s control § Customer Satisfaction impacted by: Performance Quality Availability ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 7

LTE Qo. E- Qo. S - KPIs MAPPING Network Performance Customer Experience Qo. E Application Layer (integrity, accessibility, retainability) ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom Qo. S Upper Layers (IP/UDP, IP/TCP) (throughput, delay, Layer 2&3(MAC, RLC, LLC (messaging reports related to Qo. E dimensions e. g. PDP context, HO info, codec usage) KPI Physical layer (e. g. coverage, interference) 8

Factors Affecting Availability - Radio environment/coverage - Mobility issues with handovers, missing neighbours etc. giving interruptions to data sessions - Core Network - Including DNS, routing, IP, transport issues - Content Provider Servers - Capacity, resilience, redundancy issues ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 9

Factors Affecting Performance - Throughput - Low throughput creates buffering on video streaming, poor download speeds for files - Latency - Jitter and delay in packets causing interruptions with real-time UDP streams ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 10

Qo. E expectations for each OTT is different…… § Video Streaming – requires good bandwidth. Not as much in latency as long as the buffer can handle. § Online Gaming – requires good latency. Not as much in Bandwidth. § Social media – high availability is essential to keep messages/posts up-todate § Online Shopping – Security and availability is essential. Lower requirement on bandwidth and latency ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 11

Other Non Network Related Factors Affecting Quality of Experience is a function of the performance and availability of the underlying network and services, but also other factors such as: - Encoding and delivery mechanism (video/voice) - Adaptive content systems or lack of - Poorly encoded content - Devices - Poorly designed radios, CPU and memory restrictions, operating systems - Service Design - Interfaces, protocols ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 12

[ ] CASE STUDY – OTT QOE MEASUREMENT ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 13

What to consider when measuring OTT services? From the consumer side: § Can I access the service? § Are my messages sent and received in a timely fashion? § Do I receive notifications correctly? § Is the audio quality good? From the operator side: § What is the uptime of the service? § How fast are messages delivered? § What average MOS/POLQA do customers get in a voice call? ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 14

ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 15

Whatsapp case study: Ping r. whatsapp. net Whatsapp signon https: //r. whatsapp. net DNS lookup r. whatsapp. net We can test the Qo. E of whatapp services with real devices: 1. DNS lookup 2. Ping 3. Whats. App signon But is this enough for an operator? ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 16

Whatsapp case study User Device Mobile network operator Whatsapp Clients RAN Core network Whatsapp Server The whatsapp OTT value chain § How can an operator ensure whatsapp service delivery over the mobile network? Remember: § RAN and Core Network are controlled by the operator § User device and whatsapp servers are outside operator’s control ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 17

Whatsapp case study: 24 by 7 monitoring and location of errors Cell colour indicates average error DNS Ping Whatsapp 1 2 3 Locations 4 1 Clients 2 3 Different locations or access methods ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 4 Vertical red means general failure Horizontal red means error at a specific location Different resolution actions in each case 18

Whatsapp case study: Results OTT “value chain” Core network Whatsapp. S ervice By understanding expected behavior Monitor Master can identify and alert on failures DNS time is the time to find the IP address of the server Ping time is the time to locate the server Whats. App time is the time to connect and login to the server Sign on time to Whats. App application ~0. 2 s ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 19

Whatsapp case study: Pinpointing problems Correlation between Ping and Logon times shows when an issue is outside the control of the operator ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 20

Facebook “Be a subscriber” using test equipment to measure real experience This example demonstrating a test to perform a FB update and measure time taken and outcome ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 21

Facebook benchmarking between carriers Facebook Internet Carrier 1 Carrier RANs Complex multi party interactions Carrier 3 Carrier 2 Clients TEMS Monitor Master is used to actively test Facebook interactions for different carriers Facebook “value chain” Different aspects of the service interaction must be supported by the carrier network Interactions include logon, messaging, content upload, homepage display Results used to improve core network and interworking connection Results also important for marketing ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 22

SOLUTION OVERVIEW TEMS Monitor Master probes set up with scripts to test Facebook interactions Active testing access using multiple different carriers Tests simulate different clients such including PC and Smartphone and use different bearers Tests exercise all the important functionality of Facebook from the user’s perspective Engineering detail gives radio and IP trace to help pinpoint where issues occur Multiple reports configured specifically to meet the customer’s requirements and add value ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 23

Detailed results and Value Facebook Carrier RANs Reports shows average performance of key Facebook interactions for different carriers Facebook “value chain” Shows at a glance how different carriers are performing Shows how different aspects of the Facebook service perform Results used to support marketing campaigns Detailed results used to troubleshoot errors and improve MTTR for the owning carrier Differences between bearer also used to troubleshoot Different clients also used to give further insight into performance ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 24

ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 25

Case Study - Video test per geographic location Non compressed video for regular users during off peak hours Compressed video for regular users during peak hours APN 1 GGSN 1 APN 2 GGSN 2 … … APN X GGSN Y 3 rd party video streaming service 3 rd party live TV 3 rd party subscription TV Compressed video for “all you can eat data” users at all times TEMS Monitor Master is used to actively test video streaming services Content delivery network must deliver video with different encoding schemes correctly Access point names (APNs) must deliver different service correctly Mobile network must deliver from geographical locations 24 x 7 Video Compression applied in different scenarios must be within certain Qo. E ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 26

Case Study - Video compression per subscriber profile 3 rd party video streaming service Video CDN Non compressed video for regular users during off peak hours Video optimisation and load balancing Compressed video for regular users during peak hours 3 rd party live TV 3 rd party subscription TV Compressed video for “all you can eat data” users at all times TEMS Monitor Master is used to actively test video streaming services Content delivery network must deliver video with different encoding schemes correctly Access point names (APNs) must deliver different service correctly Mobile network must deliver from geographical locations 24 x 7 Video Compression applied in different scenarios must be within certain Qo. E ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 27

Video Quality Comparison: different compressions Reports show key video quality KPIs for different compression methods Video size is reduced for compressed videos Content Delivery Network Video content Clients MOS is lower for compressed video as expected, but still within acceptable ranges Stalling/re-buffering % is similar for both compressed/non-compressed videos Video CDN value chain ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 28

[ ] CONCLUSION ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 29

Conclusion § Think about how users interact with OTT services, and what factors affect their experience § Identify methods of measuring those factors in the OTT delivery chain, especially the areas within operator’s control § Quantify and measure those factors through active testing § Act to improve those factors and measure impact ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 30

[ ] Q&A ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 31

[ ] THANK YOU! ITU Athens Qo. S and Qo. E of OTT 2015 © Ascom 25 YEARS OF INNOVATION 32