proto DUNE activities proto DUNE data management project

proto. DUNE activities • proto. DUNE data management project – This is a follow on to 3/15 workshop which culminated in a design document. This is a project to implement such. • SAMDUNE project – As a component of data management • proto. DUNE WMS – Topic of 7/28 -7/29 workshop; discussion will follow • Workers: Igor, Steve, Gabriele, Robert, Stu • Objectives: – Show that the FIFE tool set (SAM, FTS) meets needs – Produce a demo system with all components (next slides) – Eventually move these components into production 1 8/11/2016 SPPM-WG on proto. DUNE

The context • The SP-proto. DUNE detector* may produce up to 3 PB of data in a 60 -day running period in 2018 – Instantaneous data rates approach 20 Gbit/s – Average (over beam spill) rates ~ 4 Gbit/s à Multiple simultaneous streams à Large numbers of files in transit • Will have complexities (slow controls, beam info) of any large experiment * There are 2 proto. DUNE detectors, dual-phase (DP) NP 02 and single-phase (SP) NP 04 in a charged-particle beam at CERN. We focus on the SP detector. Both test full-size components of final DUNE detector. 2 8/11/2016 SPPM-WG on proto. DUNE

The components • The data path consists of: – Event builder (and possibly aggregator) processes • and associated metadata sources – DAQ buffer (currently conceived as a clustered file system supporting xrootd) – CERN central EOS – CERN Castor – Fermilab d. Cache/enstore (and other non-CERN sites) – SAM server and database – (Fermi-) FTS server, responsible for DAQ to EOS – FTS server, responsible for EOS to Castor – FTS server, responsible for EOS to Fermilab +… 3 8/11/2016 SPPM-WG on proto. DUNE

The demo systems • Using Fermi. Cloud to provision instances of the components • As we don’t have “real” DAQ systems, DAQ buffer, EOS, or Castor then we will emulate – For DAQ systems, scripts to generate files (and metadata) – For DAQ buffer, node with local file system fronted by xrootd server • eventually server such nodes fronted by xrootd redirector – – 4 For EOS, similar to DAQ buffer For SAM create an instance in the SAM development system Don’t yet know how to emulate Castor Several instances of FTS servers 8/11/2016 SPPM-WG on proto. DUNE

Issues to address • • • 5 What is the metadata (for SAM) and how is it created? How does FTS know that files are ready to be moved? How are checksums best implemented? What are the EOS interface details? What is the interface to Castor? How do the components fit into the CERN infrastructure? 8/11/2016 SPPM-WG on proto. DUNE

Effort and schedule estimates • This is not foreseen as a complex problem requiring significant integrated effort – Largest chunk of effort is. 5 FTE-year of Igor • But effort is composed of small pieces of several people, so progress is not necessarily fast or coherent • We need to get external information (metadata decisions, EOS and Castor details, …) • Igor has a charter and preliminary WBS for the SAMDUNE component – Software changes to FTS and SAM – Support for commissioning and integration 6 8/11/2016 SPPM-WG on proto. DUNE

Status and outlook • VMs for the components are created • SAM database created • Establishing xrootd servers to emulate DAQ, EOS components • Configuring FTS servers • Using proto. DUNE Redmine pages to collect Q&A on all of the details • Would like to have complete demo system in time for DUNE collaboration meeting (9/12 -9/15) – With sufficient instructions to propagate to CERN environment 7 8/11/2016 SPPM-WG on proto. DUNE