Ocean Observatories Initiative Data Management Organization NSF Data
Ocean Observatories Initiative Data Management Organization NSF Data Management Review April 28, 2014
Operating and Maintaining the OOI The OOI team will execute processes, procedures, work instructions to meet operational requirements and deliver data products. Activities include: Marine hardware and instrumentation System status monitoring, reporting and corrective actions Pre-deploy (test, integration, calibrate), Deploy (platform ops, command/control) Vehicle ops, alarms/alerts, sampling strategies, data QC Recovery, post-recovery (refurb, calibrate), data recovery Performance monitoring and reporting, Asset management, Incident Reporting Data delivery (system services and products) Cybersecurity, System monitoring, status display, software maintenance Performance measurement, network and application enhancement Hardware/Software – Development, Maintenance, Quality assessment Maintenance and improvement of data products, algorithms, metadata Data QA/QC enhancements, user request and support 1 NSF Data Management Review April 28, 2014
Anticipated O&M Organization Structure Sustaining end-to-end data delivery Ocean Leadership Safety Assurance WHOI Global and Pioneer Observatory Director Marine Assurance UCSD Cyberservices OSU Endurance Subaward SIO Global 2 NSF Data Management Review April 28, 2014 UNOLS OOSC Award Administrator UW Regional /Cabled Assets
Assessment and improvement of OOI data delivery UNOLS OOSC OOI Operations Observatory User groups sustaining data delivery Technical feasibility Technical ‘refresh’ Scheduling of approved work The OOI team sustains the approved configuration External science recommendations data QC prioritization of sensor improvements Will be integrated into the Annual Work Plan Subject to budgetary approval by NSF 3 NSF Data Management Review April 28, 2014
Operational structure: support external science and data delivery Ocean Observing Science Committee (OOSC) of UNOLS fills external advisory role. Observatory User Groups will possess expertise to inform the advisory committee. The cross-observatory science team (small in 2015) will be the internal resource for User Groups and external advisors. Size of science and data QC team evaluated annually. 4 NSF Data Management Review April 28, 2014
Organizational and Functional Management Ocean Leadership OOI-wide Data Management Team Leverage individual IOs to manage functional areas Data Mgmt Monitoring CI CG EA RSN EPE 5 NSF Data Management Review April 28, 2014 Engineering Deployment Refurbishment
Organizational and Functional Management Ocean Leadership CI CG EA RSN EPE 6 OOI-wide Data Management Team Leverage individual IOs to manage functional areas Monitoring NSF Data Management Review April 28, 2014 Engineering Deployment Refurbishment
Data Management Ocean Leadership Observatory Director OOI-wide Data Management Team Data Management Lead (TBD) – 1 FTE Will direct the activities of representatives from each Marine Organization 2015 – 3 FTE 2016 – 3 FTE 2017 – TBD following internal and external assessment Concept of Operations establishing efficiencies in non-data areas, resulting in opportunities to increase staffing within data management and data QC 7 NSF Data Management Review April 28, 2014
Objectives for data processes in O&M Sensor data will be validated through pre- and post-deployment calibrations, in-water validations when possible, and tracking of sensor engineering data during deployment. Full metadata description will be provided for each unit. After commissioning of elements of the OOI arrays, the data products will have quality control annotations (‘flags’) based on community standard error-detection algorithms. The need for additional post-processing ‘data QC’ will be assessed on an annual basis, within the constraints of the OOI operational budget. All the data QC steps will be documented and accessible to users. 8 NSF Data Management Review April 28, 2014
Maintaining Data Integrity Two aspects to Data Integrity (as reviewed in previous presentations) • Management – Observatory configuration (asset tracking) – Metadata • Subset of asset tracking • Calibration tables • Quality – QC, identifying defects of products • Automated QC flags, manual review of data – QA, identifying defects in process 9 NSF Data Management Review April 28, 2014
Pre-Deployment Procedures 1. Incoming Inspection • Completed for all Instruments and Platforms • Verifies configuration and state as delivered 2. Calibration Records • Records for each instrument or platform are archived in Vault 3. Quality Conformance Tests (QCT) • Completed for all Instruments and Platforms • Confirms basic functionality (“bench test”), detects failures or damage 4. Requirements Verification • Completed for each instrument type or Class • Validate first article against requirements and specifications 5. Platform Integration and Test • Platform operation verified using platform controller • End-to-End communication verified, instrument to shore station 10 NSF Data Management Review April 28, 2014
Maintenance of data delivery in O&M Annual Review of: – Data delivery • % of sensors providing data (weekly compilation of status) • Timeline of QC stats per sensor (monthly compilation) • Identification (weekly) and annotation (monthly? ) of data issues – – – – Data Product List Data Production Algorithms QA/QC Algorithms Data Calibration/Validation Procedures Sampling Strategy Plan Community feedback about data products, QC Propose improvements, implement through annual O&M work plan 11 NSF Data Management Review April 28, 2014
Data processes in O&M Objectives for Data QC after acquisition • Compilation of error statistics for each data product • Qualitative examination of the data records by project personnel • Aggregate performance metrics for each type of sensor and set of data products • Quarterly report of statistics provided to Observatory Director, then to External Advisory Committee and Observatory User Groups – consisting of an observatory performance evaluation and recommendations for improvements • feedback will be solicited from the user community – via UNOLS OOSC - to supplement the assessments from the OOI O&M team. 12 NSF Data Management Review April 28, 2014
Data processes in O&M Objectives for Data QC after acquisition Annual ‘data QC’ meetings will be held with NSF, the Observatory Director, the Cross-observatory Science Team and the OOSC • Establish clear recommendations for data QC improvements • Propose to implement within Work Plan (and budget) for the next period of performance. 13 NSF Data Management Review April 28, 2014
14 NSF Data Management Review April 28, 2014
- Slides: 15