Cern VMFS Best Practice to Consolidate Global Software
Cern. VM-FS – Best Practice to Consolidate Global Software Distribution Catalin CONDURACHE, Ian COLLIER STFC RAL Tier-1 ISGC, Taipei, March 2015
Outline • • Introduction Brief history Cern. VM-FS deployment at RAL EGI Cern. VM-FS Task Force EGI Cern. VM-FS infrastructure Cern. VM-FS impact on user communities biomed and Cern. VM-FS ISGC 15, Taipei, March 2015
Introduction – Cern. VM-FS ? • Read-only, globally distributed filesystem, optimized to access experiment software and conditions data • Built using standard technologies (fuse, sqlite, http, squid and caches) • Files and directories are hosted on standard web servers and mounted in the universal /cvmfs namespace ISGC 15, Taipei, March 2015
Introduction – Cern. VM-FS ? • Software needs one single installation, then it is available at any site with Cern. VM-FS client installed and configured • Repository meta-data and files are downloaded on demand locally cached ISGC 15, Taipei, March 2015
Brief History – Cern. VM-FS • Last 5 years distribution of experiment software and conditions data to WLCG sites has been changed massively – Cern. VM-FS became the primary method – No need of • local installation jobs • local SW tags (!) • (heavy loaded) file servers – It relies on a robust decentralised network of repositories replicas ISGC 15, Taipei, March 2015
Brief History – Cern. VM-FS • In parallel the use of Cern. VM-FS in communities outside WLCG has been increasing steadily – Growing number of repositories and Cern. VM-FS servers around the world • Over 2 years ago RAL Tier-1 started a non-LHC Stratum-0 service ISGC 15, Taipei, March 2015
Non-LHC Cern. VM-FS Deployment at RAL • Supported by Grid. PP UK project • Initially (Dec’ 12) for UK VOs, then extended to international small VOs (2+ NGIs) • Local installation jobs used to automatically publish the Stratum-0 – now obsolete • Replaced by web interface used by VO Software Grid Managers to upload, unpack tarballs and publish – now less used ISGC 15, Taipei, March 2015
Non-LHC Cern. VM-FS Deployment at RAL • Recently Grid Security Interface (GSI) added to transfer and process tarballs and publish – Faster and easier – Allows a programmatic way to transfer and process the tarballs – Initially based only on DN access, then VOMS Roles access also enabled – January 2015 – Software available at WN level after max 3 hours ISGC 15, Taipei, March 2015
Non-LHC Cern. VM-FS Deployment at RAL • At the moment 21 repos (~500 GB) are hosted and published at RAL • Separate Stratum-1 service for non-LHC VOs – Initially shared the WLCG Stratum-1 – Recently deployed HA 2 -node cluster – plans for a consolidated Stratum-1 at RAL (WLCG and non-WLCG) ISGC 15, Taipei, March 2015
GSI Interface Software Installation Mechanism at RAL /home/augersgm /home/biomedsgm. . /home/t 2 ksgm DN credentials VOMS Role credentials VO Software Grid Managers rsync CVMFS Uploader GSIssh/scp /cvmfs/auger. egi. eu /cvmfs/biomed. egi. eu. . /cvmfs/t 2 k. egi. eu Stratum-1@RAL Stratum-1@NIKHEF replication Stratum-0 Stratum-1@ASGC Stratum-1@TRIUMF ISGC 15, Taipei, March 2015
EGI Cern. VM-FS Task Force • EGI initiative to establish a Cern. VM-FS infrastructure that allows EGI Virtual Organizations to use it as a standard method of distribution of their software at grid sites • Kick-off meeting – August 2013 • Members: representatives from NGIs and other collaborations, VO SGMs, system admins at sites ISGC 15, Taipei, March 2015
EGI Cern. VM-FS Task Force • To promote the use of Cern. VM-FS technology amongst user communities • To create a network of sites providing Cern. VM-FS services (Stratum-0, Stratum-1, Squid proxies) • To encourage cooperation with other organizations (OSG, WLCG, regional grid infrastructures) by cross -replicating repositories for VOs supported by multiple collaborations ISGC 15, Taipei, March 2015
EGI Cern. VM-FS Infrastructure • The model follows the Cern. VM-FS topology for WLCG ISGC 15, Taipei, March 2015
EGI Cern. VM-FS Infrastructure • ‘relaxed’ topology • Stratum-0 are disjoint and represent the source repositories where software is installed by VOs • Stratum-0, Stratum-1 can be geographically colocated or not • Stratum-1 can replicate an entire Stratum-0 or can only partially replicate – the ’relaxed’ model • It partially makes use of the existent hierarchy of proxy servers used for LHC software distribution ISGC 15, Taipei, March 2015
EGI Cern. VM-FS Infrastructure Proxy Hierarchy Stratum-0 NIKHEF nikhef. nl Stratum-1 NIKHEF Proxy Hierarchy Stratum-1 CERN Stratum-0 RAL egi. eu Stratum-1 RAL Stratum-0 DESY desy. de Stratum-1 DESY Proxy Hierarchy ISGC 15, Taipei, March 2015
EGI Cern. VM-FS Extended Infrastructure Proxy Hierarchy Stratum-0 NIKHEF nikhef. nl Stratum-1 NIKHEF Proxy Hierarchy Stratum-1 CERN Stratum-0 RAL egi. eu Stratum-1 RAL Stratum-0 DESY desy. de Stratum-1 DESY Stratum-1 ASGC Proxy Hierarchy Stratum-1 TRIUMF ISGC 15, Taipei, March 2015
EGI Cern. VM-FS Infrastructure • 31 software repositories currently hosted – – HEP and non-HEP Stratum-0 s at RAL, NIKHEF, DESY 5 repos at the time of kick-off BIG change in two years! ISGC 15, Taipei, March 2015
EGI Cern. VM-FS Infrastructure • ‘egi. eu’ domain – Its importance recognized at the Cern. VM-FS WG meeting (March 2014) – Taken into account by latest Cern. VM-FS packages releases • public key and Cern. VM-FS domain configuration now part of standard installation (as for ‘cern. ch’) – cvmfs-keys v 1. 5 • no longer CERN-centric configuration – cvmfs v 2. 1. 20 • new puppet module available ISGC 15, Taipei, March 2015
Cern. VM-FS Impact on User Communities • Encouraged by the expansion of Cern. VM-FS infrastructure, more VOs have identified Cern. VMFS as a facility that suited better their needs – HEP (auger, t 2 k. org, hyperk. org, glast. org) – Space, Natural and Life Sciences (biomed, km 3 net. org, enmr. eu, pheno) – EGI fedcloud community – exploratory discussions • But software distribution to grid and cloud sites still an obstacle for some non-LHC VOs ISGC 15, Taipei, March 2015
Cern. VM-FS Impact on User Communities • Deployment issues – VOs are lacking manpower to progress more – Technology and its advantages are understood, just they do not have time – Subgroups within same VO are using subsets of software at specific sites • coordination needed to consolidate software under single Cern. VM-FS space – Sometimes VOs wait too long for sites to help them with initial tests before rolling out more widely ISGC 15, Taipei, March 2015
biomed and Cern. VM-FS • biomed - large scale international and multidisciplinary VO supporting communities from the Life Sciences sector • Divided into three sectors: medical imaging, bioinformatics and drug discovery • Member of the Life Science Grid Community – Other members: vlemed, lsgrid, vo. eu-decide. eu, enmr. eu • Openly accessible to academics and to private companies for non-commercial purposes ISGC 15, Taipei, March 2015
biomed and Cern. VM-FS • biomed - Status at September 2013 – ~80 supporting sites – ~50 sites running significant production – 3 M jobs during Sep’ 13 • Many biomed sub-groups installed various software packages at supporting sites – ‘biomedsgm’ installation jobs – Need for NFS (or AFS) space at sites – Distribution of new SW releases was time consuming ISGC 15, Taipei, March 2015
biomed and Cern. VM-FS • Timeline – Sep’ 13 – initial discussions (EGI TF Madrid) – Oct’ 13 – Mar’ 14 • • • Stratum-0 repository at RAL local install jobs used initially for upload web upload interface and GSI interface tested work to prove the concept within biomed communities first test jobs run at RAL using Cern. VM-FS space – Jan’ 14 – Apr’ 14 • supporting sites asked (GGUS, personal contact) to enable biomed ISGC 15, Taipei, March 2015
biomed and Cern. VM-FS • Timeline – Jul’ 14 • GSI as standard method to upload • biomed shifters invited to use Cern. VM-FS – Sep’ 14 • GGUS campaign for more sites to enable Cern. VM-FS support – Nov – Dec’ 14 • 44 out of 90 sites provide Cern. VM-FS • VO-biomed-CVMFS tag used to identify supporting sites • biomed community informed about Cern. VM-FS availability ISGC 15, Taipei, March 2015
biomed and Cern. VM-FS • Current status – 75 sites running biomed jobs – 90 out of 169 CEs advertise VO-biomed-CVMFS tag ISGC 15, Taipei, March 2015
Summary – Cern. VM-FS • Success story • Critical service for WLCG – soon to become critical for others • Non-LHC Virtual Organizations are successfully following the model – biomed, enmr. eu, auger – increasing number of EGI VO repos – 75 sites running biomed jobs ISGC 15, Taipei, March 2015 – 90 out of 169 CEs advertise VO-biomed-CVMFS tag
Thank you! (more) Questions? ISGC 15, Taipei, March 2015
- Slides: 27