Update of 1618 Document Pascal Ballester Pipeline Day

Update of 1618 Document Pascal Ballester Pipeline Day 2019 -11 -15 15 November 2019

The Purpose of 1618 Document The purpose of the 1618 document (ESO-037611) is fourfold: 1) It defines the science operation software components to be delivered by the instrument consortia for integration in the Data Flow System, in accordance with the ESO Observatories Operations Model (La Silla – Paranal Telescopes and ELT). 2) It defines the structure of the documentation required at the different milestone reviews PDR, FDR, PAE, PAC. 3) It provides a template development schedule for the organization of instrument projects. 4) It sets the implementation standards for the software, data, and configuration files to be delivered by the instrument projects. The document does not define the ESO Observatories Operations Model, it describes it in order to justify the requirements on the deliverables. 1618 has been applicable to every instrument project since VLT 1 st light. Version 1 in 1999 for VLT first-light, version 2 in 2004 (CPL), version 3 in 2011 (Reflex), 1 st Draft v 4 distributed to ELT consortia in 2016, together with Statements of Work. 15 November 2019

Integration of the ELT Requirements • The ELT operations model assumes that the ELT will be a facility built, managed and operated entirely by ESO, and that science operations of the ELT and the VLT will be performed in an integrated environment (ELT TLR-198) • Dataflow related developments for La. Silla Paranal and ELT instruments take place in the framework of a combined VLT/ELT data flow integration • Title of the document was generalised: ESO VLT/VLTI -> ESO Observatories • Retained the overall layout of the version 3 document, in order to facilitate comparisons and access to information • Updates planned in 1618 integrated in the ELT INS So. Ws • 1 st Draft v 4 distributed to ELT consortia in 2016 • New recommendations used for PDR documentation (e. g. Critical Algorithms sections) • Final step will be to adapt projects to the released 1618 v 4 following a Change Request 15 November 2019

Standards Review Process • 1618 is an ESO Engineering Standard • Updates of ESO Standards follow the process of the ESCB (Engineering Standards Control Board) • 1 st Draft v 4 distributed to ELT consortia in 2016, together with Statements of Work • New technologies were defined (e. g. Web-based Obs. Preparation) • Presentations to ESCB in 2016 -09 and in 2019 -04 • Draft distributed for ESO internal review in 2019 -05 • Two updates in 2019 -07 and 2019 -09 • Final RIX meetings in 2019 -10 • Draft for release provided to consortia in 2019 -11 • Completion expected by the end of the year 15 November 2019

Main Changes • Applies uniformly the standard data format for science data products (DMO Science Data Products Standard) • As a baseline, the implementation library will be the Common Pipeline Library, complemented with a collection of High-Level Data Reduction algorithms (HDRL) • The Critical algorithms shall be provided in prototype form prior to implementation, usually at the time of FDR. • Simulated raw and reduced data products, as well as a Data Reduction Chain Skeleton, shall be provided by the projects according to a schedule defined at FDR. • The Web-based implementation standard for Observation Preparation Tools replaces the deprecated Tcl/Tk implementation standard. Like for the Web -based ETCs, most of the functionality is integrated by ESO based on the specifications provided by the consortia. • Early delivery of Instrument Packages for will be required for configuring p 2 and ETCs. 15 November 2019

Applicable and Reference Documents • Guideline for Applicable Documents is to limit to what is strictly necessary for providing the deliverables: • Data formats for raw, processed data, and configuration files • All Applicable and Reference documents (as well as Linked Documents) are made publicly available. • The version of Ads, RDs is not explicitly mentioned, but set at So. W time. • Later on, in principle a Change Request is needed for each version update, sometimes the process can be lighter (in case of minor update) 15 November 2019

High-Level Data Reduction Library • Careful algorithms evaluation and verification is performed at each release. • Users may propose new algorithms if needed. • Clear and detailed algorithm requirements definition with iterative refinements. • Attentive design to allow code sharing and easy maintenance. • Extended unit and regression tests to verify results. • Code profiling, speed-up, and use of Open. MP to take advantage of multi-core architectures. • Continuous integrations tests to monitor code builds, portability, compiler warnings, documentation, static checks, and coverage. 15 November 2019

SDPs and Post-FDR Deliveries • As of FDR and in the early phase of development of the Data Reduction Software, a number of deliverables and simulated data and data products shall be provided as intermediate deliverables. • A typical timeline for the delivery of these tools is to deliver the prototype of Critical Algorithms at FDR, simulated data 6 -months after FDR, and the DRS Chain Skeleton 12 -months after FDR. Other timelines can be agreed with ESO in the So. W or at PDR, e. g. delivery at FDR • The Consortium shall provide at FDR a running prototype of the critical algorithms (key functions) of the reduction software that have been agreed at PDR. While the usage of CPL/HDRL is recommended for prototyping the critical algorithms, the consortium may choose a different implementation language. • The Consortium shall provide at FDR a schedule for the implementation of the DRS Chain Skeleton based on CPL and HDRL, including intermediate data reduction products and final science data products compliant with the Science Data Product standard for Phase 3 process. • The implementation of the DRS chain skeleton shall implement the full chain of data products, and allow to validate both the data flow chain and the data structures of the intermediate and final science data products, up to science-grade data reduction and Phase 3 data format compliance 15 November 2019

Pipeline Development Environment • Specific versions of software, and the software components on which the development environment is based (version of OS, configuration control, version of CPL, etc. . ) evolve with time. • 1618 avoids being too specific about details of the development environment. Revert 1618 v 4 to the v 3 formulation “The Data Reduction Library (. . . ) will be placed by ESO under version control”. • Updates of the operational software environment are controlled by ESO configuration boards. • The evolution of the development environment is planned within the DMO Programme, taking into account the requirements needed for operational pipelines, VLT and ELT pipeline developments. 15 November 2019

Web-based P 2 for VLT 15 November 2019

Web-based ETCs and Observation Preparation - Specified by consortia at PDR/FDR, integrated by ESO - Interfacing of instrument specific modules - Maintenance: infrastructure vs. specialised modules 15 November 2019