Forschungszentrum Karlsruhe in der HelmholtzGemeinschaf t Forschungszentrum Karlsruhe

Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaf t Forschungszentrum Karlsruhe Gmb. H, Institut für Wissenschaftliches Rechnen, Postfach 36 40, 76021 Karlsruhe Bruno Hoeft, Marc Garcia Marti LHC-Optical. Private. Network NETWORK AT GRIDKA 10 Gbit LAN/WAN evaluations NIC – tuned for throughput Chelsio German / European 1) IP via MPLS Tunnel International IP RWTH-Aachen Uni-Dresden Fnal Uni-Freiburg MPI u. Uni-München SLAC Uni-Wuppertal CNAF SARA GSI / FZU Prague / Desy PSNC Poland (2007 10 Gbit) CERN (2006/7/8 (2007 1/10 Gbit) 1 Gbit) 2007 10 Gbit 240 Mbyte netto } HEP Standard mtu 8160 Bytes mtu 8180 Bytes mtu 9216 Bytes mtu Tier-0/Tier-1 (T 0/T 1) LHC-OPN Lightpath Tier-2 LHC Lightpath T 2 T 2 T 1 T 2 T 0‘ Standard mtu 8180 Bytes mtu 9216 Bytes mtu DFN X-Wi. N - PCI-X Bus: theoretical limit 8. 5 Gbps approx. - tested max. rate 7. 8 Gbps (linux kernel packet generator) 10 GE AR DWDM 10 GE FZK-Internet Router … Cache Headnode Login. Server ALICE/Atlas/CMS/LHCb - Chelsio 0. 020 ms -The throughput of the Chelsio NIC is still normal after tuning for latency - Intel 0. 016 ms - The throughput of the Intel NICs is neglectable after tuning for latency d dcache 2) pools … router private network FS … Compute. Node SAN … … switch … FS FS FS NAS … NIC – tuned for Latency LCG(se/ce/ui) LCG(RGMA/MON/BDII) LCG(RB/LFC/DC-SRM) … Ethernet 1 Gbit Fiber. Channel 2 Gbit Ethernet 10 Gbit T 0 DWDM lightpath Backup (end 2006) T 1 DWDM lightpath Bandwidth 10 GE T 2 DWDM lightpath Bandwidth 1/10 GE T 2 L 2 -IP-Tunnel (MPLS) Login. Server Babar/CDF/DZero/Compass … FZK-LAN dcache 2) pools intern Standard mtu 8160 Bytes mtu 8180 Bytes mtu 9216 Bytes mtu Standard mtu (server side) 8160 Bytes mtu (server side) 8180 Bytes mtu (server side) 9216 Bytes mtu (server side) Firewall extern Standard mtu 8180 Bytes mtu 9216 Bytes mtu Standard mtu (server side) 8180 Bytes mtu (server side) 9216 Bytes mtu (server side) Grid. Ka-Internet Router 10 GE … Intel Compute. Node WAN setup -- Grid. Ka - CERN Cisco 12416 GEANT Juniper M 160 DFN Cisco 12410 Juniper M 160 Routing: - External Routing via BGP -e. BGP with the Forschungszentrums AS-No. : 34878 exterior Boarder Gateway Protocol -i. BGP synchronising the e. BGP router interior Boarder Gateway Protocol - Internal redundance secured via OSPF - Layer-2 tunnel for distributed Storage Server via Policy Based Routing - Private network space separated into switched VLans Cisco 6509 Cisco 7609 Gridka (Karlsruhe) CERN oplapro 96 WAN throughput 10 gtk 113 / TCP cloning Window size 14. 0 MByte Tx. Int. Delay=500 Rx. Int. Delay=500 Window size 2. 0 MByte Tx. Int. Delay=500 Rx. Int. Delay=500 Window size 8. 0 MByte Tx. Int. Delay=250 Rx. Int. Delay=250 Window size 8. 0 MByte Tx. Int. Delay=74 Rx. Int. Delay=74 Window size 8. 0 MByte Tx. Int. Delay=15 Rx. Int. Delay=15 Mmrbc set to 512 bytes Mmrbc set to 2048 bytes Mmrbc set to 4096 bytes 1 GE versus 10 GE Server Nodes Total Cost of Ownership (TCO): - Dramatically reduced costs (per NIC/high end server) - the troughput increased from 2. 8 Gbit to 7. 0 Gbit with TOE (standard MTU) - eight 1 GE standard server = one 10 GE Server - higher administration needs to be taken into account - PCI-Express will allow for the full rate of 10 GE - 250/6000 Euro per 1 GE/10 GE router port Email: Bruno. Hoeft@iwr. fzk. de; Marc. Garcia@iwr. fzk. de Phone: +49 7247 82 8172 Web: www. gridka. de www. fzk. de/iwr 1) „Connecting WLCG Tier-2 Centres to Grid. Ka“ of Dr. Andreas Heiss presents more details about the at Grid. Ka assigned Tier-2 sites 2) Dr. Doris Ressmann will present in her talk more detailed information at the d. Cache setup of Grid. Ka. „using TSM to create a high performance Tape connection“ Track: Computing Facilities and Networking, Monday 14: 20 -14: 40