Status and Directions Patrick Fuhrmann On behave of

Status and Directions Patrick Fuhrmann On behave of the project team Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 1

Reminder : The collaboration HTW Berlin (University) Dmitry Al Vincent Gerd Anupam Femi Jürgen Marina Paul Sibel Tigran And Patrick Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 2

General principles • d. Cache collaboration avoids dominant partner – Focus on requirements of partner customers – Common design meetings – Cross partner code reviews • Finding collaborators on selected topics – – EU Projects (see later) German national projects (LSDMA, Joint German Russian Initiative) Teaching at HTW Berlin (UNI) FUSE integration with KI, DUBNA • Focus on standards whenever possible – E. g. data access and AAI – If not possible, find standardization bodies • RDA working group on Qo. S • SNIA on CDMI Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 3

Collaborations and projects Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 4

H 2020 Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 5

EU H 2020 Projects • INDIGO Data. Cloud – will finish end of Sep 17 – Achievements • Implementation of AAI and Qo. S in d. Cache. • EOSC Pilot – Jan 17’ until End of 18’ – Goal : Getting scientific communities into the cloud Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 6

On projects with external entities • New Proposal in INFRA 21 – e. Xtreme Data Cloud (XDC) lead by CNAF • The e. Xtreme Data. Cloud (XDC) project will develop scalable technologies for federating storage resources and managing data in highly distributed computing environments. Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 7

XDC WP 4 mostly d. Cache and CERN • T 4. 1: Intelligent dataset distribution and data lifecycle management based on policies at the site and the federated level. This includes long term supervision of data, aka “Data Life Cycle Management”. In a second step, data location, and in consequence data movement decisions can be based on feedback from access pattern probes, metadata and infrastructure conditions, which could be storage system load, component failures and network parameters. • T 4. 2: Smart Caching. This task is a special case of T 4. 1. Data is temporary migrated close to the geographical location it is needed to be processed. Storage space availability and usage pattern considerations are controlling automatic cleaning of less popular datasets. Technologies will be provided to install and operate cache only systems as satellites of the persistent federated storage infrastructure • T 4. 3 Orchestrating Computing Workflows based on policy driven or adaptive data movements. The federated storage infrastructure will provide information on data ingestion and data movement to the central orchestrator which in return can launch arbitrary compute process on the data. In the easiest case, metadata extraction can be triggered on data ingest or pre-processing initiated on data arrival at a computing facility. The actual steering is performed by a workflow engine within the INDIGOData. Cloud Paa. S Orchestrator component. Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 8

Project with the Russian Federation Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 9

Projects with Russia • Joint Research Project (Germany - Russia) – – Title: “Federated data prediction framework” Russia (NRC “Kurchatov Institute”, JINR in Dubna, PNPI (Gatchina) ) Germany (DESY (d. Cache. org), LMU (Belle II), HTW Berlin ) 3 Years with about 3 FTE per country. • WLCG and XFEL Building a large scale federation – Project with NRC “KI” and JINR (Dubna) • Alexei Klimentov, Andrey Zarochentsev and Andrey Kirianov – In preparation: FUSE for d. Cache via the xroot protocol • Implemented by Ivan Kadochnikov (visited DESY for 4 weeks) Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 10

Strategic directions Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 11

Strategic directions Focusing of the support for Large Computing Infrastructures Scientific Community requirements Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 12

Large Computing Infrastructures • High availability – Dark Computer centre – Non-disruptive updates – Both in 3. x • Improved multi-site security (Federations) – Secured control connections • Allowing to efficiently use available underlying storage technologies. (Trying not to reinvent the wheel) – CEPH, GPFS, Cloud (S 3) …. – All kind of mixtures Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 13

Improved fine grain security Local Computer Center Core Remote SSL Core SSL Satalite Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 14

You’ll learn about … Authentication Protocol Engines Basic / SAML / Open. ID Connect NFS/p. NFS grid. FTP x. Rootd/d. Cap Virtual File-system Layer Direct Attached http(s) based Web. DAV SRM REST Qo. S Moni Orchestration Resilient Mgr External Storage Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 15

Support for modern scientific communities • Quality of Service in Storage – Media type resp. Media quality specification on ‘store’ • Based on importance of data and budget – Media transitions (manual or automatic) • Bring online • After data quality check – Data Life Cycle support Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 16

Simple Data Life Cycle Use-Case XFEL - DB-like access - Mutable Files HDF 5 Immutable Fast Access (multiple Copies) Archive Fast Analysis Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 17

Extended Data Life Cycle Use. Case XFEL - DB-like access - Mutable Files Immutable (multiple Copies) Fast Analysis Different Geographical Location HDF 5 Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 18

Support for modern scientific communities • Adding support for standards – Data Access • Available : Http/Web. DAV, NFS 4. 1, Grid. FTP • Recently added : Samba (for XFEL users) and FUSE – AAI and Portal support • Open ID Connect • Macaroons for delegation Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 19

Support for Portals (Macaroons) Restricted Macaroon - Lifetime - IP Number - ETC Super Macaroon Access based on limited macaroon (no login into d. Cache) Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 20

Sync and Share chown chmod acl set /afs/cern. ch/home Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 21

Sync and Share and Qo. S My Home ( ) Scratch Regular Permanent ( ) Archive Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 22

Simplifying d. Cache operations Consolidating d. Cache control access GUI Media Control (Qo. S) d. Cache admin REST CDMI Command Line REST Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 23

Consolidated GUI Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 24

“Federations” Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 25

Hope to learn about Federations at this meeting. Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 26

Finish further reading www. d. Cache. org Status and Plans | WLCG 2017, St Petersburg | Patrick Fuhrmann | 11 July 2017 | 27