http www canarie ca CAnet 4 International Grid

http: //www. canarie. ca CA*net 4 International Grid Testbed Bill. St. Arnaud@canarie. ca Tel: +1. 613. 785. 0426

Problem 1. TCP throughput over long fat pipes very susceptible to packet loss, MTU, TCP kernel, Buffer memory, trail drop, AQM optimized for commodity Internet, etc 2. Packet loss can result from congestion, but also underlying BER – achieve a gigabit per second with TCP on a coast-to-coast path (rtt = 40 msec), with 1500 byte packets, the loss rate can not exceed 8. 5 x 10^-8 packets – “End to end” BER for optical networks 10^-12 to 10^-15 which means packet loss rate of approximately 10^-8 to 10^-11 – The bigger the packet the greater the loss rate!!! 3. Cost of routers significantly greater than switches for 10 Gbps and higher (particularly for large number of lambdas) 4. Lots of challenges maintaining consistent router performance across multiple independent managed networks – MTU, auto-negotiating Ethernet, insufficient buffer memory 5. Require consistent and similar throughput for multiple sites to maintain coherency for grids and SANs and new “space” storage networks using erasure codes e. g. Oceanstore 6. For maximum throughput OS and kernel bypass may be required 7. Many commercial SAN/Grid products will only work with Qo. S network

Possible Solutions 1. 2. 3. For point to point large file transfer a number of possible techniques such as FAST, XCP, parallel TCP, UDP, etc – Very scalable and allows same process to be used for all sorts of file transfer from large to small – But will it address other link problems? Datagram Qo. S is a possibility to guarantee bandwidth – But requires costly routers and no proven approach across independent managed networks (or campus) – Does not solve problem of MTU, link problems, etc E 2 E lightpaths - all solutions are possible – Allows new TCP and non TCP file transfers – Allows parallel TCP with consistent skew on data striping – Allows protocols that support OS bypass, etc – Guarantees consistent throughput for distributed coherence and enables news concepts of storing large data sets in “space” – Uses much lower cost switches and bypasses routers

What are E 2 E lightpaths? > Customer controlled E 2 E lightpaths are not about optical networking – E 2 E lightpaths do not use GMPLS or ASON > The power of the Internet was that an overlay packet network controlled by end user and ISPs could be built on top of telco switched network – CA*net 4 is an optical overlay network on top of telco optical network where switching will be controlled by end users > More akin to MAE-E “peermaker” but at a finer granularity – “Do you have an e 2 e lightpath for file transfer terminating at a given IX? Are you interested in peering with my e 2 e lightpath to enable big file transfer? ” – Lightpath may be only from border router to border router > With OBGP can establish new BGP path that bypasses most (if not all) routers – Allows lower cost remote peering and transit – Allows e 2 e lightpaths for big file transfer

e 2 e Lightpaths Of elephants and mice Small mice traffic is routed over normal IP path Normal IP/BGP path Only y. y. y. 1 advertised to x. x. x. 1 via OBGP path Only x. x. x. 1 advertised to y. y. y. 1 via OBGP path Optical “Peermaker” x. x. x. 1 OBGP path y. y. y. 1 Application or end user controls peering of BGP optical paths to set up dedicated route for transfer of elephants

CA*net 4 Edmonton Saskatoon Calgary Vancouver Winnipeg Halifax Regina St. John's Victoria Montreal Ottawa Seattle Minneapolis CA*net 4 Node Fredericton Toronto Boston Existing CA*net 4 OC 192 Trans. Light OC 192 Charlottetown Chicago New York Halifax

Canada sets land speed record Vancouver <-> Geneva CAN ARI 2 x. G b E Nethe E circ uits Sta r. Lig ht SUR F 2 x. G net www. i. Grid 2002. org for more info on i. Grid 2002 circu b. E its r. Ligh t

SAN land speed record VANCOUVER OTTAWA 8 x GE @ OC-12 (622 Mb/s) CHICAGO 1 x GE loop-back on OC-24 Sustained Throughput ~11. 1 Gbps Ave. Utilization = 93%

ATLAS/CMS: Data Grid Hierarchy Low level Trigger data ~PByte/sec Online System Experiment ~100 -1500 MBytes/sec Tier 0 +1 ~2. 5 Gbps Tier 1 IN 2 P 3 Center Tier 3 INFN Center RAL Center ~2. 5 Gbps Tier 2 Institute ~0. 25 TIPS Physics data cache Workstations CERN 700 k SI 95 ~1 PB Disk; Tape Robot Institute 0. 1– 10 Gbps Tier 4 FNAL: 200 k SI 95; 600 TB 2. 5 Gbps Tier 2 Center Tier 2 Center

International Grid Testbed > Joint CERN, SURFnet, STAR LIGHT, Trans. Light project > Objectives: 1. To validate and test software for customer control and routing of lightpaths 2. Test remote processing of of low level trigger data from the ATLAS test beam. 3. Develop and adapt grid applications which are designed to interact with a Light. Path Grid Service which treats networks and network elements as grid resources which can be reserved, concatenated, consumed and released. 4. Characterize the performance of bulk data transfer over an end-to end lightpath. 5. To investigate and test emerging technologies and its impact on high speed long distance optical networks. These technologies include 10 Gbit Ethernet, RDMA/IP, Fibre Channel/IP, serial SCSI, Hyper. SCSI over long distance ethernet, etc. 6. Collaborate with the EU ESTA project which is developing 10 Gb. E equipment with CERN, industrial and other academic partners.