Sea Data Net tools session IFREMER Sea Data

Sea. Data. Net tools session IFREMER Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Agenda • Improving the work flow from data centres to CDI catalogue using the Sea. Data. Net tools • Presentation and Hands-on session on tools – NEMO: last developments and corresponding exercises – OCTOPUS: New software developed in Sea. Data. Cloud – MIKADO: last developments and corresponding exercises sdn-userdesk@seadatanet. org – www. seadatanet. org

Improving the workflow using the Sea. Data. Net tools NEMO, OCTOPUS, MIKADO Michèle Fichaut - Ifremer Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Challenge for a data infrastructure • Distribute homogeneous data and metadata files to data users in standardized formats that can be used in standard tools – Available tools • NEMO • OCTOPUS • MIKADO sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 NEMO • Tool for data file conversions Data files at any ASCII format sdn-userdesk@seadatanet. org – www. seadatanet. org Data files at Sea. Data. Net format

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 NEMO • This the aim of NEMO software – Generate files at Sea. Data. Net formats • Ocean Data View (ODV, text csv format) • Net. CDF (Binary format) • Med. Atlas (Text format, enriched in metadata) – Sea. Data. Net formats because • Data files contain specific Sea. Data. Net information like LOCAL_CDI_ID, EDMO_CODE • Mapping to P 01 (parameters), P 06 (units), L 22 (instruments), CSR id…. sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Exemple of specific SDN information • SEMANTIC header • <subject>SDN: LOCAL: PRES</subject><object>SDN: P 01: : PRESPR 01</obj ect><units>SDN: P 06: : UPDB</units><instrument>SDN: L 22: : TOOL 0035</ instrument> • <subject>SDN: LOCAL: TEMP</subject><object>SDN: P 01: : TEMPPR 01</o bject><units>SDN: P 06: : UPAA</units><instrument>SDN: L 22: : TOOL 0035 </instrument> • <subject>SDN: LOCAL: DOX 2</subject><object>SDN: P 01: : DOXMZZXX</o bject><units>SDN: P 06: : KGUM</units> • LOCAL_CDI_ID=FI 35201500250_00012_H 10 • <sdn_reference xlink: href="http: //seadata. bsh. de/cgicsr/XML/xml. Download_V 2. pl? edmo=486&identifier=15002500 " xlink: role="is. Observed. By" link: type="SDN: L 23: : CSR"/> sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 OCTOPUS (1) • New Sea. Data. Cloud Tool for Sea. Data. Net data file checks and conversions Data files at another Sea. Data. Net format Data files at one Sea. Data. Net format Format checks sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 OCTOPUS (2) • Tool for Sea. Data. Net data file splitting Multi-station file at one Sea. Data. Net format LOCAL_CDI_ID 1 LOCAL_CDI_ID 2 … … LOCAL_CDI_IDn Mono-station file at the same or another Sea. Data. Net format LOCAL_C DI_ID 1 … … LOCAL_ CDI_ID 2 LOCAL_ CDI_IDn … sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 MIKADO • Generation of standardised metadata XML descriptions for Sea. Data. Net catalogues Metadatabase Automatic ISO-19139 / ISO 19115 Metadata descriptions Manual sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 MIKADO • • • EDMED : description of Marine data sets EDMERP : Description of Marine projects CSR : Cruise summary reports EDIOS : Description of observatories CDI : description of data – Vertical profile (point) – Time series (point) – Trajectories, tracks (curves or surface) sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Why again a workshop on MIKADO and NEMO? • These tools are in used since the beginning of Sea. Data. Net more than 10 years ago • Still number of data and metadata files contain errors – Some of them are BLOCKING ERRORS – Others are just TIME CONSUMING MISTAKES when performing the generation of the products • Aim of this presentation sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Example of a Sea. Data. Net ODV file sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Example of a Sea. Data. Net ODV file Comment lines - Optional sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Example of a Sea. Data. Net ODV file Sea. Data. Net external links - Optional sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Example of a Sea. Data. Net ODV file Sea. Data. Net Semantic header - Mandatory Parameter mapping between local parameter name and NVSV 2 vocabulary sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Example of a Sea. Data. Net ODV file ODV Column headers Fixed labelling for the first columns from “Cruise” … to “Bot. Depth” Column for quality flags mandatory after each parameter column: labelled QV: SEADATANET sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Example of a Sea. Data. Net ODV file Station header ODV Data Lines sdn-userdesk@seadatanet. org – www. seadatanet. org Measured parameters and QC flags

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Example of a Sea. Data. Net ODV file • • ODV Column headers (CH)and Semantic headers (SH) have to be coherent Number of SDN parameter mapping lines = Number of measured parameters Label in the CH= label in SDN: LOCAL in SH P 01 Parameter code of SH match local label of CH P 06 Unit code in SH match the unit of the CH sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Different error types (1) Error in Sea. Data. Net semantic header Wrong Right Example Which tool to avoid or correct Wrong P 01 parameter code PH ZZXX PHXXZZXX NEMO to avoid Use of deprecated parameter code MMUSHGXX D 2930561 OCTOPUS to correct File with the V 1 NERC vocab terms P 011 P 01 P 061 P 06 OCTOPUS to correct Semantic header missing Non SDN format Errors in the mapping lines like missing brackets sdn-userdesk@seadatanet. org – www. seadatanet. org NEMO, to avoid • /units> • </units> NEMO, to avoid

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Different error types (2) Error in ODV metadata column headers Wrong labelling of the metadata column headers (which is fixed) Wrong Right Example 1. Bot. depth [m] Bot. Depth [m] 2. yyyy-mm-dd. Thh-mm-ss. sss YYYY-MM-DDThh: mm: ss. sss 3. Latitude [deg. North] Latitude [degrees_north] sdn-userdesk@seadatanet. org – www. seadatanet. org Which tool to avoid or correct NEMO to avoid OCTOPUS to correct (1 and 2)

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Different error types (3) Error in ODV data column headers Example Which tool to avoid or correct Missing space between parameter label and parameter unit in column header DOXY[ml/l] DOXY [ml/l] NEMO to avoid Incorrect primary variable (first measured variable after Bot. Depth) For sediment core: Depth below sea surface: ADEPZZ 01 Depth below surface of the bed: COREDIST For vertical profile: Sequence number: ACYCAA 01 Depth below sea surface: ADEPZZ 01 Pressure: PRESPR 01 For Time series: Data and time ISO: DTUT 8601 NEMO to avoid Extra last empty column NEMO to avoid, OCTOPUS to correct sdn-userdesk@seadatanet. org – www. seadatanet. org QV: Sea. Data. Net column missing NEMO to avoid

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Different error types (4) Error in ODV metadata and data lines Wrong Right Example Which tool to avoid or correct Wrong EDMO_code EDMO of the originator EDMO of the CDI_Partner NEMO to avoid Missing value set to a default value in stead of empty 999. 99 NEMO to avoid But ODV understand Missing QC flag set to 9 for missing value No Flag NEMO to avoid NO QC flag or Wrong scale of QC Sea. Data. Net QC flag scale has to Flags be used (NVS list L 20) sdn-userdesk@seadatanet. org – www. seadatanet. org NEMO to avoid

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Different error types (5) Incoherencies between data files (ODV, Net. CDF, Med. Atlas) and metadata files (CDI XML) Wrong Right Example Which tool to avoid or correct List of P 02 parameters in the CDI XML files does not match with the P 01 described in the data files linked to variable measured parameters from one station to another in a cruise file In CDI XML: P 02 = TEMP, PSAL, NEMO => to NTRA, NTRI avoid In data files only: nutrients P 01 codes, no temperature, salinity In CDI XML: P 02= only NTRA and NTRI No match between LOCAL_CDI_ID in CDI XML and data file 123456789_ABCD in data file 123456789 -ABCD in CDI XML Should be the same Checker in the cloud CDI partner EDMO_code different Originator EDMO code in data file Checker in in CDI xml and in data file in stead of CDI_partner EDMO the cloud code sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Tools in Sea. Data. Net context Sea. Data. Net Vocabulary V 2 Meta. Data in DBMS New metadata Metadata in Excel files MIKADO Metadata XML ISO 19115 ISO 19139 CSR ISO 19139 MIKADO EDIOS ISO 19115 Coupling table Data in DBMS New data Data in ASCII files Format X NEMO EDMED ISO 19115 OCTOPUS sdn-userdesk@seadatanet. org – www. seadatanet. org Data centres EDMERP ISO 19115 ad lo er wn ag Do an M Collection of ASCII files Format ODV, Net. CDF, Med. Atlas Local copy of data for downloading CDI ISO 19139 Data request AAA Downloading RSM S E A D A T A N E T P O R T A L European portal

Tools in Sea. Data. Cloud context MARIS NVS BODC Vocabulary CDI central catalogue Data centres New metadata Metadata in en Excel file Metadata Meta. Data MIKADO XML in DBMS ISO 19139 Coupling table Data in ASCII files Format X RM Collection of ASCII files Format ODV, Net. CDF, Med. Atlas Local TEMP copy of the corresponding unrestricted and restricted data files at all possible SDN formats RM Local archive of restricted data files with version number, at all possible sdn-userdesk@seadatanet. org – www. seadatanet. org SDN formats OCTOPUS Data request Downloading CDI ISO 19139 Import MIKADO Data in DBMS New data Replication Manager (RM) SDN user interface RSM Manager (IM) Marine_ID EUDAT Cloud copy of unrestricted data with version number at all possible SDN formats TEMP copy of restricted data with version number at formats asked by user RM

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Some examples of data flows using the tools Octopus Example 1 Files SDN format No QC Files format X Example 2 Files format X Example 3 Files format X QC QC Files SDN format QC Local DB Metadata QC Mikado Local DB CDI central Metadata CDI-Summary + Data Files SDN format QC sdn-userdesk@seadatanet. org – www. seadatanet. org Octopus RM - IM EUDAT CLOUD RM - IM

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Advantages of using the tools • Sure to be standard • Lots of errors avoided • Further standard and format evolutions are taken into account in the tools without any changes on your side • No incoherencies between allowed – Data file (ODV, Net. CDF, Med. Atlas) – XML CDI description – Coupling table sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Call for beta testers • Most of the tools are first tested internally by the developers and then by the responsible of the specifications – Vanessa for MIKADO – Michèle for NEMO and OCTOPUS • Risk of missing important bugs – Need volunteers for beta testing sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 sdn-userdesk@seadatanet. org I agree to be a beta tester of Software_name sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Help needed for the training session • Some of the exercises may be trivial for some of you who are expert users of the tools Help welcome to assist the less experienced users! sdn-userdesk@seadatanet. org – www. seadatanet. org

Sea. Data. Cloud 1 st training session, Ostende, Belgium, 20 -27 June 2018 Any questions? sdn-userdesk@seadatanet. org – www. seadatanet. org