KISTIGSDC SITE REPORT Jeongheon Kim Duseok Jin On

KISTI-GSDC SITE REPORT Jeongheon Kim, Duseok Jin On the behalf of KISTI GSDC 18 April 2016 HEPi. X Spring 2016 Workshop DESY Zeuthen, Germany

2016 -04 -18 CONTENTS • KISTI GSDC Overview • Tier-1 operations • GSDC System & Plan HEPi. X Spring 2016 Workshop 2

2016 -04 -18 HEPi. X Spring 2016 Workshop KISTI GSDC OVERVIEW 3

2016 -04 -18 HEPi. X Spring 2016 Workshop KISTI Location 4 Daedeok R&D Innopolis South Korea Seoul Rare Isotope Accelerator (To be constructed) 2 h by car Daejeon Daegu Gwangju Busan Jeju Island KISTI 30 Government Research Institutes 11 Public Research Institutes 29 Non-profit Organizations 7 Universities

2016 -04 -18 HEPi. X Spring 2016 Workshop 5 KISTI GSDC KISTI (Korea Institute of Science and Technology Information) • Government funding research institute for IT founded in 1962 • 600 people working for National Information Service(distribution & analysis) , Supercomputing and Networking • Operating Supercomputing and NREN Infrastructure Supercomputer: 307. 4 TFlops at peak(14 th ranked at Top 500 in 2009; 201 st now) • NREN Infrastructure: KREONet 2 • • • Domestic: Seoul ←(100 G)→ Daejeon International: HK ←(100 G)→ Chicago/Seattle (Member of GLORIAD) GSDC (Global Science experiment Data hub Center) • Government funding project to promote research experiment providing computing power and storage HEP: ALICE, CMS, Belle, RENO • Others: LIGO, Bioinformatics • • Running Data-Intensive Computing Facility 13 staffs: sysadmin, experiment support, external-relation, administration • Total 7, 900 cores, 6, 000 TB disk and 1, 500 TB tape storage • History of GSDC ALICE T 2 operation start 2007 Formation of GSDC ALICE T 2 Test-bed 2009 KISTI Analysis Facility 2010 2011 ALICE T 1 Test-bed GSDC Facility Full T 1 for ALICE 2012 ALICE T 1 candidate 2013 Start CMS T 3 2014 2015 10 Gbps LHCOPN Established

2016 -04 -18 HEPi. X Spring 2016 Workshop 6 GSDC System Overview Private Public 4 Spine switches 74 Leaf switches 500+ Servers in 22 racks 14 Storage racks 4 tape frames HITACHI HNAS EMC ICILON 1. 5 PB § Torque/MAUI § 4, 500 slots § ALICE T 1, Belle, RENO § HTCondor § 2, 500 slots § CMS T 3, LIGO, KIAF 1. 5 PB IBM TSM/GPFS 3. 5 PB HITACHI USP/VSP EMC Clariion/VNX

2016 -04 -18 HEPi. X Spring 2016 Workshop TIER-1 OPERATIONS 7

2016 -04 -18 HEPi. X Spring 2016 Workshop 8 Jobs Stable and smooth running Participating in HI Reco Run at the end of 2015 261 M jobs done for the last 6 months Maximum 3, 402 concurrent jobs = 37 k. H 06 (84 nodes x 32 (logical) cores + 20 nodes x 40 (logical) cores) KISTI, 3. 19% = 2. 6 M jobs for the last 6 months ~ 3300 ~ 2500 ~ 1300 Sep 2015 Maximum 1, 344 concurrent jobs (84 nodes x 16 cores) > 5 GB memory available per job For HI Reco Run ~ Feb 2016 Mar 2016

2016 -04 -18 HEPi. X Spring 2016 Workshop Storage 1. 465 PB ALICE RAW Data Transferred 9 last 6 months 99% Storage Availability for the ← May 2013 Jun 2013 3 Yrs Use History May 2015 1, 465 TB Used (Tape) 855. 6 TB Used (Disk) Tape Disk Run 2 Data Taking 400 TB Disk buffer for helping fast access to data archived 99% Storage Availability of R/W for last 6 months

2016 -04 -18 HEPi. X Spring 2016 Workshop 10 Site Availability/Reliability • 99. 8% Reliable for the last 6 month (from Sep-2015 to Feb-2016 ) • Monthly Target for Reliability of ALICE test: 97% • Less than 10 days of yearly downtime • On track for a stable and reliable site • Participating in weekly WLCG operations meetings(1 times (Mon) per week): reporting operation-related issues • 24/7 monitoring & maintenance contract • 2 persons responsible for on-call < Monthly Availability/Reliability (%) > Sep Oct Nov Dec Jan Feb Reliability 100 100 100 99 Availability 100 100 95 100 99

2016 -04 -18 HEPi. X Spring 2016 Workshop 11 KISTI-CERN Network (LHCOPN) 10 Gbps Upgrade done by 31 st April 2015 Nether. Light @Amsterdam KRLight @Chicago GSDC/KISTI @Daejeon URFnet) (S SURFnet Seattle C+PC-1) SKB(EA SKB(Le vel 3) SKB(Level 3) SKB(EAC+UNITY) LA SURFnet (CANARI E) SURFnet t) (SURFne New York CERN @Geneva NET KREO SK Broadband (~’ 16. 8) SURFnet

2016 -04 -18 HEPi. X Spring 2016 Workshop 12 Performance • > 9 Gbps peak (~ 1 GB/s) observed CERN→KISTI (5 min) MRTG @ MX 960 • CERN IT provided a gateway, 500 parallel transfers • xrd 3 cp crashed with Xrootd v 3. 3. 4 (fixed @ v 4 or later) • Max 1 GB/s peak @ alimonitor. cern. ch • Confirmed full capacity CERN IT Gateway Multi-stream: 500 Max peak: 1 GB/s KISTI-GSDC 10 G enabled Average: 65 MB/s alimonitor. cern. ch

2016 -04 -18 HEPi. X Spring 2016 Workshop GSDC SYSTEM 13

2016 -04 -18 HEPi. X Spring 2016 Workshop 14 Controlled by Puppet and Foreman • Profile implemented by the Puppet • Role defined in the Host Group of Foreman

2016 -04 -18 HEPi. X Spring 2016 Workshop Central Authentication by IPA • User • DNS • Kerberos • SUDO rule • Automount • Host based access control 15

2016 -04 -18 HEPi. X Spring 2016 Workshop Log Collecting by ELKStack • Messages • IPTables • SNMP trap • Torque job 16

2016 -04 -18 HEPi. X Spring 2016 Workshop Plan • Additional resources will be procured • ~720 CPU cores • ~2 PB disk storage (NAS) • ~1. 5 PB tape storage • Linux container based provisioning system will be deployed for supporting several researches 17

2016 -04 -18 HEPi. X Spring 2016 Workshop 18 o. Virt Node o. Virt Engine k 8 s UI k 8 s Dashboard k 8 s Kubedash Grafana In. Flux. DB Kibana Elasticsearch Fluentd Gluster Experimental Setup Kubernates Cluster k 8 s controller manager k 8 s scheduler k 8 s podmaster etcd k 8 s proxy hyperkubelet Docker Flanneld monit Atomic Host k 8 s apiserver

2016 -04 -18 HEPi. X Spring 2016 Workshop Persistence Data Storage 19

2016 -04 -18 Install Script HEPi. X Spring 2016 Workshop 20

2016 -04 -18 HEPi. X Spring 2016 Workshop Running Screenshot 21

Master Plan for Container based Provisioning System Monitoring Analyzing Logging ETC Services HTCondor Torque MPI Mesos Cluster Docker Datacenter OS (Apache Mesos with Kubernetes) Node OS Bare metal Node OS VM Cluster Node OS External Cloud