How is the Internet Performing Les Cottrell SLAC

How is the Internet Performing? Les Cottrell – SLAC Lecture # 2 presented at the Workshop on Scientific Information in the Digital Age: Access and Dissemination ICTP, Trieste, Italy October , 2009 www. slac. stanford. edu/grp/scs/net/talk 09/ictp-perform. ptt 1

Overview • Internet characteristics – Users, capacities, satellites, packet sizes, protocols, routing, flows • How is it used apps etc. • How the Internet worldwide is performing as seen by various measurements and metrics • Application requirements • Comparisons with Development Indices 2

USERS 3

1500 Millions of Users 0 200 400 600 N America Australia Europe L America M East Asia Africa World 1000 Millions of users Asia Europe N Amer L Amer Africa M East Austrlasia Internet Usage growth ‘ 95 -’ 10 500 Penetration % 0 20% 50% 80% 95 Year 00 05 094

Example: China • China not connected to the Internet until May 1994 – 1 st permanent IHEP/Beijing used satellite via SLAC – www. computerworld. com. au/article/128099/china_cel ebrates_10_years_being_connected_internet 5

• Internet city connections 2. 8% growth/year ~¼ world pop uses Internet Developed world saturating Developing catching up 73% penetration US 43% users from Asia Where are they Internet Users 2002 6

Capacities 7

What have they got? Capacity From Telegeography 8

Min RTT (ms) Who is still on Satellite GEOS (Geostationary Earth Orbit Satellite) good coverage, but expensive in $/Mbps Terrestrial 9 broadband costs 50 times that in US, >800% of monthly salary c. f. 20% in US AND long delays min RTT > 450 ms, usually much larger due to congestion Easy to spot Clear signature

Packet sizes & types 10

Packet size • primarily 3 sizes: WHY? Measured Feb 2000 at Ames Internet e. Xchange Cu, mulative probability % close to minimum=telnet and ACKs, 1500 (max Ethernet payload, e. g. FTP, HTTP); ~ 560 Bytes for TCP implementations not using max transmission unit discovery Mean ~ 420 Bytes, median ~ 80 Bytes Packets ~ 84 M packets, < 0. 05% fragmented Bytes Packet size (bytes) 11

Internet protocol use • There are 3 main protocols in use on the Internet: SLAC protocol flows ICMP TCP Flows/10 min In Out – UDP (connectionless datagrams, best effort delivery), – TCP (Connection oriented, “guaranteed” delivery in order) – ICMP (Control Message protocol) TCP dominates today UDP Time Feb-May 2001 12

Routing 13

Hops • Hop counts seen from 4 Skitter sites (Japan, S. Cal, N. Cal, E. Canada, i. e. 10 -15 hops on average RTT Weak RTT dependence on hop count 95% 50% 5% Hops Hop Count 14

Richness of connectivity • Angle = longitude of AS HQ in whois records • Radius=1 -log(outdegree(AS)+1)/(maxoutdegree + 1) – Outdegree = number of next Hops As’ accepting traffic • Deeper blue & red more connections • All except 1 of top 15 AS’ are in US, exception in Canada • Few links between ISPs in Europe and Asia 15

Today’s routing less via US www. nytimes. com/2008/08/30/business/30 pipes. html • Invented in US • 1 st 30 yrs most traffic thru US • 70%=>20% in 10 yrs • No central control • Patriot act=>store info outside US • China, India, Japan making larger investments • More level playing field • Harder for CIA! 16

Routes are not symmetric Advanced to U. Chicago RTT ms • Min, 50% & 90% RTT measured by Surveyor • Notice big differences in RTTs • May be due to different paths in the 2 directions or to different loading U. Chicago to Advanced 17

Flows 18

Flow sizes SNMP Real A/V AFS file server Heavy tailed, in ~ out, UDP flows shorter than TCP, packet~bytes 75% TCP-in < 5 k. Bytes, 75% TCP-out < 1. 5 k. Bytes (<10 pkts) UDP 80% < 600 Bytes (75% < 3 pkts), ~10 * more TCP than UDP Top UDP = AFS (>55%), Real(~25%), SNMP(~1. 4%) Can roughly characterize as power law with slope & intercept 19

Flow lengths TCP outbound flows Measured by Netflows tied off at 30 mins Active time in secs • 60% of TCP flows less than 1 second • Would expect TCP streams longer lived – But 60% of UDP flows over 10 seconds, maybe due to heavy use of AFS at SLAC – Another (CAIDA) study indicates UDP flows are shorter 20 than TCP flows

Applications 21

Usage • P 2 p hit by RIAA law suits • Moving to video, social networking – Video on demand double/2 years ’ 08 -’ 13 • i. Phones (only peripherally a phone) – Mobile traffic doubles each year Yahoo Google You. Tube Facebook 22

How it is used & when asert. arbornetworks. com/2009/08/theinternet-after-dark Enterprise & tier 1 23

Web use characteristics • Size of web objects varies from site to site, server to server and by time of day. – Typical medians in 2000 varied from 1500 to 4000 bytes • Also varies by object type, e. g. medians for – movies few 100 KB to MBs, postscript & audio few 100 KB, text, html, applets and images few thousand KB • Size of average web page tripled in 5 years 2003 -2008 • www. websiteoptimization. com/speed/tweak/av erage-web-page/ Bytes 24

Why increasing • New users (easier for user, more coverage) • New apps: You-Tube, climate modeling … • New tools: manual(hand tuned) Automatic generation – Web 2: Ajax, Javascript, CSS • Broadband more elaborate/attractive designs possible • desktop to web apps • e. g. mail, calendars, photo albums, games. . . 25

Impact on backbones: e. g. Current and Historical ESnet Traffic Patterns ØESnet Traffic Increases by 10 X Every 47 Months, on Average Apr 2006 1 PBy/mo. Nov 2001 100 TBy/mo. July 2010 10 PBy/mo. Terabytes / month Jul 1998 10 TBy/mo. 53 months Oct 1993 1 TBy/mo. Aug 1990 100 MBy/mo. 40 months 57 months 38 months Log Plot of ESnet Monthly Accepted Traffic, January 1990 – December 2008 26

Performance by Metric 27

What does performance depend on? • End-to end internet performance seen by applications depends on: – round trip times – packet loss – jitter – reachability – bottleneck bandwidth – implementation/configurations – application requirements • Data transmitted in packets 28

RTT from SLAC to the World msec. RTT ~ distance/(0. 6*c) + hops * router delay Router delay = queuing + clocking in & out + processing ITU G. 114 300 ms RTT limit for voice 2/3 countries of world Ok for voice, rest mainly in Africa What is the problem with > 300 ms? 29

Brazil 300 ms E. Coast Europe & S. America RTT (ms) E. Coast US W. Coast US Frequency RTT from California to world Europe 0. 3*0. 6 c 300 ms Longitude (degrees) Source = Palo Alto CA, W. Coast WHY these distributions? RTT (ms. ) 30 Data from CAIDA Skitter project

Jitter • • • Variability of RTT, many ways to measure “Jitter” = IQR(ipdv); ipdv(i) =RTT(i) – RTT(i-1) Usually at edges, so ~distance independent Impacts smooth flows e. g. Vo. IP, video, real-time Haptics (surgery) < 1 ms; H. 323 <40 ms with buffer Can improve voice with de-jitter buffer, e. g. 70 ms to smooth the flow But…. Internet Jitter seen from SLAC to World Sep’ 0831

Losses • On good lines usually congestion • Wireless d. B loss, net devices • Usually last mile • Distance independent • Big effect • Realtime, games, Voice, typing echo • 1% loss Vo. IP annoying 32

Derived Throughput Central Asia, and Africa are in Danger of Falling Even Farther behind In 10 years at the current rate Africa will be 1000 times worse than Europe 1993 Behind Europe 5 Yrs: Russia, Latin America, Mid East 6 Yrs: SE Asia 9 Yrs: South Asia 12 Yrs: Cent. Asia 16 Yrs: Africa Derived throughput ~ 8 * 1460 /(RTT * sqrt(loss)) Mathis et. al 33 3

Where is best Throughput? 34

Voice over IP • Affected by: – Loss, RTT, Jitter, • Quality measured by Mean Opinion Score (MOS) – Can convert from RTT, loss & jitter to MOS – MOS values: 1=bad; 2=poor; 3=fair; 4=good; 5=excellent. – Typical reasonable range for Voice over IP (Vo. IP) is 3. 5 to 4. 2. – Russia and L. America improved dramatically in 2000 -2002 as moved from GEOS to terrestrial. – US, Europe, E. Asia, Russia and the M East (all above MOS = 3. 5) good. S. E. Asia marginal, S. Asia need a lot of patience – C. Asia and Africa are pretty much out of the question in general. 35

Application requirements • Based on ITU Y 1541 & Stanford (Haptics) Application Real time Vo. IP WAN Web free Stream Haptics (remote connectivity services video surgery) 1 way delay 150 ms 1000 ms undefined 400 ms 160 ms ‘jitter” 50 ms 1000 ms undefined 17 ms 1 ms Loss 10 -3 undefined 10 -5 0. 1 10 -3 • The Vo. IP loss of 10^-3 used to be 0. 25 but that assumed random flat loss – actual loss is often bursty • Tail drop in routers • Sync loss in circuits, bridge spanning tree reconfiguration, route changes 36

Compare with Development Indices Abv. Name Organization Gross Domestic Product per CIA capita HDI Human Development Index UNDP GDP Countries Date of Data 229 2001 -2006 175 2004 DAI Digital Access Index ITU NRI World Economic Forum 120 2007 UNDP 72 1995 -2000 ITU Transparency Organization 180 139 2004 -2005 1996 -2003 180 2007 Network Readiness Index Technology Achievement TAI Index DOI Digital Index OI Index CPI Corruption Perception Index 180 1995 -2003 Choose most: up-to-date, countries, important factors HDI & DOI 37

Human Development Index • A long and healthy life, as measured by life expectancy at birth • Knowledge, as measured by the adult literacy rate (with two-thirds weight) and the combined primary, secondary and tertiary education gross enrollment ratio (with one-third weight) • A decent standard of living, as measured by GDP per capita (or Purchasing Power Parity (PPP) in US$). • http: //www-iepm. slac. stanford. edu/pingermetrics-motion-chart. html 38

HDI vs Throughput HDI 0. 4 0. 6 0. 8 1 • Dot size=population, color =region • Bottom left = bad (Africa, blue), top right=good 1000 Throughput (kbps) 10000 39

Digital Opportunity Index DOI R 2 = 0. 67 0 0. 2 0. 4 0. 6 0. 8 • DOI tracks infrastructure, opportunity & utilization • Strong Correlation, less subjective • Ping. ER throuphut Quicker/easier to update cf DOI Throughput (kbps) 40

What’s next • Mobile devices • 40 G (trans. Atlantic, US) & 100 Gb backbones • On demand dynamic dedicated services (layers 1 & 2) – Reserve a path at some bandwidth for some time – Use Qo. S to deliver – HEP, Radio Astronomy, climate research • IPv 6 41

Questions & more study • www. internetworld. stats. com • www-iepm. slac. stanford. edu/pinger • www. slac. stanford. edu/comp/net/wanmon/tutorial. html • www. slac. stanford. edu/xorg/icfa-net-paperjan 09/report-jan 09. doc • Question: why a Dolphin for a Ping. ER Logo 42

IP Addresses pingable June 2003 • From CAIDA, SDSC • Grey= not allocated • Black= not pingable • Companies own class A 43

Growth 2003 -2006 • More areas allocated, • Existing areas more colorful June 2003 Nov 2006 44

• Gives quality measure • Seen from San Diego, US Skitter • Steeper = less jitter, i. e. better • Small values better Cumulative % Cumulative RTT distributions RTT ms 45

Throughput also depends on window • Optimal window size depends on: – Bandwidth end to end, i. e. min(BWlinks) AKA bottleneck bandwidth – Round Trip Time (RTT) – For TCP keep pipe full • Window (sometime called pipe) ~ RTT*BW – Can increase bandwidth by orders of magnitude Src Rcv • If no loss Throughput ~ Window/RTT K C A t = bits in packet/link speed RTT 46

Loss seen from US to groups of Sites 50% imp rove men t / ye ar ETSI DTR/TIPHON-05001 V 1. 2. 5 threshold for good speech 47

Detailed example of improvements Increase of bandwidth by factor of 460 in 6 years, more than kept pace - factor of 50 times improvement in loss Note valleys when students on vacation 48

Loss to world from US Using year 2000, fraction of world’s population/country from www. nua. ie/surveys/how_many_online/ 49

How are the U. S. Nets doing? In general performance is good (i. e. <= 1%) ESnet holding steady, still better than others Edu (v. BNS/Abilene) &. com improving 50

Losses for 28 days in May 2001 % Loss DNS Internet WWW ISP • Measured by MIDS to 583 DNS services, 383 Web services, 1367 Internet (ping) hosts, & 1225 ISPs (routers) 51

Losses between Regions 52

Bulk throughput • Important for long TCP flows where we want to copy large amounts of data from one site to another in a relatively short time, e. g. file transfer • Depends on RTT, loss, timeouts, window sizes 53

Throughput quality TCPBW < 1/(RTT*sqrt(loss)) Note E. Europe catching up Macroscopic Behavior of the TCP Congestion Avoidance Algorithm, Matthis, Semke, Mahdavi, Ott, Computer Communication Review 27(3), July 1997 54

“Jitter” from N. America to W. Europe “Jitter” = IQR(ipdv), where ipdv(i) =RTT(i) – RTT(i-1) 214 pairs ETSI: DTR/TIPHON-05001 V 1. 2. 5 (1998 -09) good speech < 75 ms jitter 55

“Jitter” between regions ETSI: DTR/TIPHON-05001 V 1. 2. 5 (1998 -09) 75 ms=Good 125 ms=Med Jitter varies with loading 225 ms=Poor 56

SLAC-CERN Jitter ETSI/TIPHON delay jitter threshold (75 ms) 57

Reachability Within N. America, & W. Europe loss, RTT and jitter is acceptable for Vo. IP But what about reachability 58

Reachability – Outage Probability Surveyor probes randomly 2/second Measure time (Outage length) consecutive probes don’t get through Heavy tailed outage lengths (packet loss not Poisson) http: //www-iepm. slac. stanford. edu/monitoring/surveyor/outage. html 59

Europe seen from U. S. Monitor site Beacon site (~10% sites) HENP country Not HENP & not monitored 200 ms 1% loss 7% loss 650 ms 10% loss 60

Asia seen from U. S. 10% loss 3. 6% loss 0. 1% loss 250 ms 640 ms 450 ms 61

Latin America, Africa & Australasia 4% Loss 170 ms 220 ms 700 ms 2% Loss 350 ms 62

Animated monthly 2000 20% loss 200 ms RTT Big is Bad 20% unreachable 63

RTT worldwide from the Matrix 64

More Information • IEEE Communications, May 2000, Vol 38, No 5, pp 120 -159 • IEPM/Ping. ER home site – www-iepm. slac. stanford. edu/ • CAIDA/Skitter home site – www. caida. org/home/ • Matrix Net home site – www. matrix. net/index. html • Surveyor home site: – www. advanced. org/csg-ippm/ 65

Country dispersion • Seen from Japan • After 3 to 4 hops most goes to US. Probes – In some cases goes US & back to jp – Some goes to UK & onto other European countries Hops 66

Route maps • Simple routes from TRIUMF, Canada to several sites already gets quite complex TRIUMF DESY UW SLAC FNAL CERN KEK 67

Getting more complex • Ping. ER Beacon sites in US seen from TRIUMF, Vancouver (from Andrew Daviel, TRIUMF) 68

Connections by country NL IT US Unknown RU UK JP DE 69

Autonomous Systems (AS) Disperson • Color indicates the AS responsible for the router at the hop, height is number of probes for that route • Seen by Skitter at Palo Alto US (F root name server) Hop number 70

Hosts by regions • Jan 2001, 109 Million hosts http: //www. internetworldstats. com/stats. htm – Source: Internet Software Consortium (www. isc. org) Notes: • see web site also for hosts/population • Many. com are in N. America • S. Asia = in (36 K), pk (6 K), lk, bd • E. Asia=jp, cn, my, sg, tw, hk, th, id, bn, mm • Mid East=il, kw, lb, ae, tr, sa • TLDs with hosts~238 • Total TLDs~258 71

Backbone utilization Shows utilization of I 2/Abilene backbone links, NB Backbone < 30% loaded Most losses at exchange points & edges 72

Typical Internet traffic by Application • CERFnet link • Dominated by WWW (http) Mail WWW FTP http: //www. cisco. com/en/US/s Real. Audio olutions/collateral/ns 341/ns 52 http: //www. datacenterknowle 5/ns 537/ns 705/ns 827/white_p dge. com/archives/2007/06/22/ aper_c 11481360_ns 827_Networking_S youtube-10 -percent-of-allolutions_White_Paper. html 73 internet-traffic/

SLAC Traffic profile Mbps in SLAC offsite links: OC 3 to ESnet, 1 Gbps to Stanford U & thence OC 12 to I 2 OC 48 to NTON HTTP Profile bulk-data xfer dominates Last 6 months Mbps out iperf 2 Days FTP SSH bbftp 74

SLAC Internet Application usage Ames IXP: approximately 60 -65% was HTTP, about 13% was NNTP Uwisc: 34% HTTP, 24% FTP, 13% Napster 75

RTT Region to Region OK White 0 -64 ms Green 64 -128 ms Yellow 128 -256 ms NOT OK Pink 256 -512 ms Red > 512 ms OK within regions, N. America OK with Europe, Japan 76

RTT(ms) RTT from Japan to world Longitude Seen from Japan 77