KOLKATA Tier2Alice Grid Kolkata Tier II ALICE and

KOLKATA Tier-2@Alice Grid Kolkata Tier II @ ALICE and Status Site Name : Tier-2 Site for the WLCG (World Wide Computing Grid) GOCDB Name: - IN-DAE-VECC-02 VO : - ALICE Group: - EHEPAG Unit: - VECC City: - KOLKATA Country : - INDIA Team: Subhasis Chattopadhyay Vikas Singhal Prasun Singh Roy Grid Computing Architecture WLCG Grid based on MONARCH Tier model T. K. Samanta and S. K. Pal helped in establishing the centre in the initial years.

KOLKATA Tier-2@Alice Grid Computing Facility at VECC Why Tier-2 India’s contribution in 2 main detectors of the ALICE PMD and Muon Arm and large volume of raw, calibrated and simulated data, therefore decided to build TIER-2. VECC, Kolkata is the only Tier-2 for the ALICE. CERN 128 Kbps Bandwidth 2002 started with 2 Computers Journey from 2002 to till now Computing Storage (2 core to 4000 (512 MB to cores) 400 TB) Bandwidth (128 Kbps to 1 Gbps) Year 2 PC 512 MB 128 Kbps shared link 2002 2 Tower Servers 40 GB as DAS 512 Kbps 2003 9 HP 1 U Servers 400 GB in HP MSA 500 2 Mbps Dedicated Link 2004 17 Wipro 1 U Single Core 2 TB Wipro NAS 4 Mbps from Bharti 2006 40 HP blades dual core 108 TB HP EVA 30 Mbps from SAN Reliance 2008 8 HP Blades Quad Core 25 TB i-scsi 100 Mbps from VSNL (ERNET) 2009 32 Dell, Dual Processor 200 TB IBM DS 5100 300 Mbps from NKN 2011 GPU Server (448 cores) 2 TB HDD 1 Gbps NKN 2012 Intel Xeon Phi 244 core 150 TB Disk Servers Requested for 10 Gbps 20142016 10 Gbps (within 2 weeks) 2017 48 Dell Servers installing 2

KOLKATA Tier-2@Alice Grid Evolution of Grid Computing Facility and Cooling Solution 2008 2006 2010 2012 - now ØHot and Cool Air is separated via Cold Aisle Containment. ØTemperature gradient between Cold and Hot aisle is 5 o. C. Cooling Solution logical diagram ØPower usage effectiveness (PUE) =Total Facility Power/ IT Equipment Power = 1200 Units / 816 Unit per Day = 1. 47 ØCooling solution reduced cooling power consumption by half. ØManagement and monitoring of the server, storage is from outside Cold Aisle Containment. 3 3

KOLKATA Tier-2@Alice Grid ALICE Job completed @Kolkata Nearly 4000000 jobs successfully completed, at Kolkata Cream Consistently every hour more than 70 Jobs successfully completing during last 6 years. No AMC for any server. Maintaining In-house only 24 x 7 Operation, 95% availability 4 Vikas Singhal, VECC, INDIA

KOLKATA Tier-2@Alice Grid Kolkata Tier-2 Resources Total : -Computing -> 448 cores (Equivalent 5 K HEPSpec 2006 Computing Resources) DELL Quad Core Blades 28*2*4*2=448(HT) Due to new hardware arrived few older nodes removed. Within a month new cores will be under production. Storage : - 174 TB (78 TB SAN Based) 1 Xrootd Redirector 1 disk Server SAN based 78 TB (IBM) No warranty for the HP EVA (72 TB) Will move this data if could recover. EOS : - 96 TB (Usable) 3 Dell Power. Egde R 730 48 TB each 1 Gbps WAN Network speed (Will increase to 10 Gbps) 10 Gbps Backbone Network 5

KOLKATA Tier-2@Alice Grid More than 2500 HT cores of Computing Resources procured § Procured 12 DELL Power. Edge FX 2 Enclosures. § Each contain 4 DELL Power. Edge FC 630 servers. § Each server configuration: 2 Nos of Intel Xeon E 5 -2680 v 4 2. 4 GHz with 14 cores 8 * 16 GB RDIMM, 2400 MT/s 960 GB of SSD harddisk. 2 * 10 Gigabit network cards. § Total cores 48*2*14*2=2688 (HT) § Installation is almost complete. § Scientific Linux CERN 6. 8 installed § 10 G network connected. § Benchmarking is going on. § Approx cost of the equipment = $ 300, 000. 6

KOLKATA Tier-2@Alice Grid 51 TFlops of Computing Resources Theoretical Peak Performance Rpeak= CPU speed GHZ * total number of core * operation/cycle Rpeak = 2. 4*28*16 gigaflop = 1075. 2 gigaflop = 1. 0752 teraflop (Single Server) Rpeak value for all 48 nodes will be Rpeak = 1. 0752*48 Tflops = 51. 6096 Tflops Preliminary HPL Benchmarking test results: - (Test completed yesterday only) is 4. 30471 e+04 Gflops or 43. 0471 Tflops. [root@wn 123 HPL]# tail -20 xhpl_intel 64_dynamic_outputs_48 nodes_log 2. txt T/V N NB P Q Time Gflops ----------------------------------------WC 00 C 2 R 2 871012 192 12 8 10233. 83 4. 30471 e+04 HPL_pdgesv() start time Mon Oct 9 22: 28: 12 2017 HPL_pdgesv() end time Tue Oct 10 01: 18: 46 2017 ----------------------------------------||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)= 0. 0023805. . . PASSED ============================== Finished 4 tests with the following results: 4 tests completed and passed residual checks, 0 tests completed and failed residual checks, 0 tests skipped because of illegal input values. ----------------------------------------End of Tests. Want to know HEP Spec 2006 number for the Intel Xeon E 5 -2680 v 4 2. 4 GHz processors with 35 MB of cache. 7

KOLKATA Tier-2@Alice Grid Physical Network Connectivity upto Kolkata red 10 Gbps a Sh bs p 0 G TEIN-3 Network s/1 bp 4 G CERN LHCONE TIFR Mumbai bp s 10 G Upgradation to 10 G Redundant Link upto Kolkata-tier 2 is about to finish. 10 G bps 1 Gbps Internet POP in Mumbai 10 Gbps s bp G 0 Core Network 1 ps 10 Gb POP in Kolkata 10 Gbps 4 -10 Gbps 1 Gbps x Gbps SINP Kolkata 10 Gbps 10 1 Gbps Gbp s VECC Kolkata 1 Gbps 10 G Switches installed at both VECC and SINP site. Kolkata Tier-2 8

KOLKATA Tier-2@Alice Grid 1 G Network Utilization by Kolkata Tier-2 during April 2017 9

KOLKATA Tier-2@Alice Grid Bandwidth Test for Kolkata Tier-2 10

KOLKATA Tier-2@Alice Grid Backbone Network inside Kolkata Tier-2 Brocade Switches • • • 10 Gb connection to each server, 40 Gb connectivity between the core switches, Both Fiber and Copper connectivity, Connection via DAC Cable. Dual path, Full Redundancy. Vikas Singhal, VECC, INDIA 11

KOLKATA Tier-2@Alice Grid Restructured Power Distribution Replaced earlier 3*40 KVA UPS due to safety measures. Procured a new 80 KVA UPS and one redundant line via 160 KVA UPS from Computer Division. . Proper planning done to avoid single point of failure. Two MCBs boxes installed. Providing UPS power to MCB Box from 2 different sources. Every network rack is getting power from two MCBs to avoid SPOF. Cabling to servers are done in such a way that every server is getting power from both MCB A and MCB B. Thanks to ELECTRICAL Section, VECC for coordinating and performing the entire work. 12

KOLKATA Tier-2@Alice Grid-Peer Tier-3 Cluster Status • • • More Load on the CLUSTER as CBM user also utilizing the cluster 8 Numbers of HP BL 675 G 7 Blade servers each with 4 * AMD Opteron Processor 6380(16 Core) 3 Number of Dell M 610 Blade servers each with 2 * Intel Quad Core E 5530 Xeon 2. 4 GHz CPU 8 MB cache and 16 GB RAM. 6 out of 8 HP blade servers are dedicated for non-interactive nodes and rest is being used for CBM work. 3 Dell blade servers are being used as interactive node. Extensively used by VECC users and PMD Collaborators, completed more than 35000 jobs successfully in last 3 months. 75 TB storage, almost filled up. 75 + active users (across India. ) 45 + active users (in VECC. ) Tape based backup of Tier-3 storage performed 13 twice in a month.

KOLKATA Tier-2@Alice Grid Connectivity between Asian Tiers Not bad increasing day by day. 10/10/2017 1

Achievements KOLKATA Tier-2@Alice Grid Milestone (Achievement) Reasoning Grid Computing Facility and Awareness. Only Center in India for ALICE. Awareness towards the High Performance Computing (HPC). Expanding the knowledge of GRID Computing and Related Technology. Achieved initial pledge in 2012. within XI plan budget (No extra) Green and Efficient cooling solution implemented. (First in Eastern India) No cooling loss. Less power required. Server efficiency increased. Implemented similar at NIBMG, DBT Successfully running for last 15 years more than 90% availability. Tier-3 Cluster for Collaborators. 100% utilized by Indian Collaborators. More than 25 Ph. D thesis completed using Grid Computing Facility of VECC. Supporting STAR, ALICE, CBM, Medical Imaging, INO Providing computing resources to all the projects. Trained more than 30 graduate student for working in HPC community. (Can Participate in National Super Computing Mission) Indian Grid Certification Authority (IGCA), Bangalore Due to requirement for ALICE only, IGCA established. Thanks to Subrata da and his team. ) Knowledge of Digital and Grid Certificate. (RA for IGCA) National Knowledge Network (NKN) India connected with the LHC- ONE Network via NKN. Asian Tier Centre Forum Networking and data exchange between Asian countries. 15

KOLKATA Tier-2@Alice Grid Future Road Map and Vision Ø Planning to collaborate with Industry. Exploring Cloud Computing via Industry partnership. (Discussion going on with Microsoft (Brij initiated the same)). Ø Parallel computing is the only solution for utilizing present computing infrastructure. Ø Accelerated or Heterogeneous Computing is new evolving field. (Participating at CBM @ FAIR in this direction). Ø Focus on High Throughput Computing (HTC) using of Accelerates like NVIDIA GPU, AMD, APU, Intel Co-processors. Ø Spreading knowledge on Parallel and Heterogeneous Computing. Ø For the huge data resources, using the low cost storage solution based on the EOS CERN. Procuring the low cost storage boxes. 16

KOLKATA Tier-2@Alice Grid Thank You 17