Data Quality Ed Chapman OOI Chief Systems Engineer

Data Quality Ed Chapman OOI Chief Systems Engineer 1 4/27/2014 1

Goal Address Areas for Recommendations: #2 “Data Policy, Data Quality Protocols and Procedures” and #4 “Data sampling rate strategy development and management ” Specific topics: “Shoreside & at-sea instrument and subsystem quality/calibration procedures/protocols, automated thresholds/flags, manual data QC, exception management, and long term time-series data sampling rate management. ” 2 4/27/2014 2

Shoreside & at-sea instrument and subsystem quality/calibration procedures/protocols 3 4/27/2014 3

Pre-Deployment Procedures 1. Incoming Inspection • Completed for all Instruments and Platforms • Verifies configuration and state as delivered 2. Calibration Records • Vendor records for each instrument or platform 3. Quality Conformance Tests (QCT) • Completed for all Instruments and Platforms • Confirms basic functionality (“bench test”), detects failures or damage 4. Instrument Requirements Verification • Completed for each instrument model • Evaluate first article against Requirements and specifications 5. Platform Integration and Test • Platform operation verified using platform controller • End-to-End communication verified, instrument to shore station • Platform Requirements Verification 4 4/27/2014 4

At-Sea Procedures: Pioneer-1 Platform monitoring • Full platform function available when within Wi. Fi range • Communication with shore station when out of range Shipboard underway sampling • Meteorology time series from Knorr bow mast • Thermosalinograph time series from Knorr system • Bathymetry from echoshounder and multi-beam Shipboard CTD profiles • Post-deployment casts at each of 3 sites for Pioneer-1 • Seabird 9 -11 with DO, Fluoro, beam x-miss, turb, PAR Physical Samples • Post-deployment casts at each of 3 sites for Pioneer-1 • Salinity and Oxygen completed onboard • Nitrate/Nitrate, Chlorophyll and Carbon system done in shore labs 5 4/27/2014 5

Automated QC Thresholds and Flags L 1 b and L 2 b 6 4/27/2014 6

Instrument Driver and Agent Permanent storage Calibration Table Data Product Algorithm Secondary Post-Deployment calibration values POLYVAL Algorithm User Secondary Post-Recovery calibration values POLYVAL Algorithm QC Flags Interpolation Lookup Tables QC algorithms (range, spike, stuck, gradient, trend, combined)

Automated QC Checks • Seven QC Checks – Global Range Test – Local Range Test – Spike Test – Stuck Value Test – Trend Test – Gradient Test – Combined QC Flags 8 4/27/2014 8

When? • QC Checks are run on a periodic basis – when data is ingested from the uncabled instruments – Continuously from the cabled instruments • QC Flags are stored. 9 4/27/2014 9

Automated QC actions – PSs create Look Up Tables and values are uploaded through the UI as csv files 10 4/27/2014 10

Automated QC Updates • If new values are uploaded for any of the QC Flags those values overwrite the original values. • OOINet reruns the QC check for all data products and creates and stores new QC Flags • QC is “value added” so we don’t retain prior flags 11 4/27/2014 11

Human in the loop QC L 1 c and L 2 c 12 4/27/2014 12

Instrument Driver and Agent Permanent storage Calibration Table Data Product Algorithm Secondary Post-Deployment calibration values POLYVAL Algorithm User Secondary Post-Recovery calibration values POLYVAL Algorithm Interpolation Lookup Tables QC algorithms (range, spike, stuck, gradient, trend, combined) Human in the loop L 1 c

Human in the Loop QC Actions • PS periodically downloads an L 1 or L 2 product • PS performs HITL QC locally on PS machine • PS uploads L 1 c or L 2 c values, and HITL metadata (provenance, etc. ) into OOINet • User who downloads L 1 or L 2 product to which HITL QC has been applied will see L 1 c or L 2 c variables in the downloaded time series – Only for the time range for which the HITL QC was applied 14 4/27/2014 14

Human in the Loop QC Updates • If new HITL values are uploaded for a time period that has already been uploaded those values overwrite the original values. 15 4/27/2014 15

Relationship of QC level a, b, and c 16 4/27/2014 16

Database L 0 L 1 Data Product Algorithm L 2 Data Product Algorithm Primary Calibration Function L 1 a Secondary Calibration Functions L 2 b L 1 b QC Algorithms Human In The Loop L 1 a L 1 b and QC flags L 1 c Human In The Loop L 0 GUI User L 2 c QC flags L 2 b

exception management 18 4/27/2014 18

Long term time-series data sampling rate management 19 4/27/2014 19

Pivotal and Default Sampling Rates • PSs and external scientists established pivotal and default sampling rates • Pivotal is the minimum necessary sampling to answer long term science questions • Default is the standard sampling rate • Rates guided by power and energy budgets for each platform type. 20 4/27/2014 20

Annual Review of Default Rate • Annual external reviews to recommend default rate changes • Annual iteration between OOI engineers and external science advisors (via UNOLS committee) to assure that recommended adjustments are feasible 21 4/27/2014 21

Adjustment to sampling rates • Any recommended change in sampling rate by external proposers will be evaluated with respect to the required maintenance of 'pivotal' rates and power and energy budgets (this becomes part of the annual interaction between OOI and the UNOLS advisory structure) • O&M team will sustain the default sampling rates. • Changes in sampling rates only occur in response to: – safety threats to life or property – technical issues that compromise performance – pre-approved responses to defined events (volcanic eruptions, hurricanes…) – approved retasking of observatory elements (NSF proposal process, UNOLS scheduling) 22 4/27/2014 22

Questions? Specific topics: “Shoreside & at-sea instrument and subsystem quality/calibration procedures/protocols, automated thresholds/flags, manual data QC, exception management, and long term time-series data sampling rate management. ” 23 4/27/2014 23