Evolution of the IOOS Glider DAC and Potential

Evolution of the IOOS Glider DAC and Potential Applications for other DMAC Realms DMAC Annual Meeting 2020 Wednesday Oct 14 John Kerfoot, Bob Fratantonio Ben La. Cour, Ben Adams, Leila Baghdad. Brahim

Goal Grow the glider community to improve the overall effectiveness and efficiency of gliders. Increase subsurface observations to meet operational requirements and forecasting needs. Mechanisms ● IOOS Glider Data Assembly Center (GDAC) ● Underwater Glider User Group (UG 2) 2

Overview: Links Providers Registration Page https: //gliders. ioos. us/providers/ Providers Status Page https: //gliders. ioos. us/status/ ERDDAP https: //gliders. ioos. us/erddap/index. html Wiki https: //ioos. github. io/ioosngdac/ Glider. Map https: //gliders. ioos. us/map/ 3

Glider Metrics for 2009 -2020 Active Deployments 4 https: //gliders. ioos. us/

What’s New? 5

What’s New? Infrastructure (backend) hardening ● Containerization ● Take advantage of cloud services ○ Modernized profile plotting ● Automated builds with Travis CI ● Service monitoring/alerting ○ Influx. DB/Grafana ● TODO: ○ Updated API (w/ Swagger) 6 ○ Update File. System (EFS)?

What’s New? Faster Access to Datasets ● Used to take up to 2 Hours for new deployment files to be accessible on ERDDAP ● Latency reduced to ~15 mins ● Testing out event driven system to remove latency ○ Should be available soon! ● ERDDAP ‘Files’ access for individual file access 7

What’s New? GTS Ingest ● Glider DAC adheres to new IOOS Metadata Profile v 1. 2 guidance on NDBC/GTS ingest ● By default, DAC deployments will add gts_ingest=True for new datasets if that global attribute is not available ● Providers may specify gts_ingest=False if they wish, or specify individual variables 8

What’s New? Reorganized Wiki ● https: //ioos. github. io/ioosngdac/ ● Wiki has been cleaned up and updated. ● Moving forward, this will be the authoritative source of Glider DAC information ● Please provide feedback! ○ 9 glider. dac. support@noaa. gov

What’s Next? ● ALL variables to be included in the DAC ● Even further reduced latency in files arriving to DAC ● Website updates ○ ug 2 ○ gliders. ioos. us ● Glider. Map enhancements ○ New layers (maritime boundaries, storm tracks, forecast models, etc) ● New provider dashboard 10

Development of Data Provider Tools/Dashboards 1. Existing System a. Deployment Registration i. Register active and delayed-mode datasets ii. Set attribution iii. Set operating institution b. Visualization i. Real-time and near real-time map of deployment/datasets tracks ii. Time-series plots of CTD parameters c. NCEI Archiving The glider DAC currently functions as ONLY a data repository. There are no feedback mechanisms/tools allowing data providers to keep tabs on the progress of the dataset once submitted to the DAC. Goal: Enable data providers to track and quantify dataset progress from operator -> DAC -> GTS and self-diagnose issues along the way.

Existing Dataset Monitoring?

New Data Provider Tools 1. System Status a. Dataset status i. WMO id? ii. Active or Completed b. Time Coverage i. Time coverage start ii. Time coverage end iii. Number of days c. Number of Profiles d. Links i. Data Access (ERDDAP) ii. Plotting (ERDDAP) iii. Deployment Registration (updating attribution, operator and dataset status) 2. Deployments a. b. c. d. e. f. g. Attribution Operating Institution Data provider Time Coverages Bounding Box Dataset Summary Dataset Imagery

Building the Displays How can we do this? 1. ERDDAP a. Simple and easy access to dataset metadata i. Advanced Search 1. Bounding Box/Geographic search 2. Time coverages ii. Query individual datasets for observations iii. Lots more! b. Simple and easy creation of imagery i. Maps ii. Profile plots iii. Time-Series 2. Python a. gdutils: https: //github. com/kerfoot/gdutils i. erddapy: https: //github. com/ioos/erddapy ii. pandas: https: //pandas. pydata. org/ iii. seaborn: http: //seaborn. pydata. org/

System Status Selection by glider, WMO ID and/or data provider https: //marine. rutgers. edu/cool/data/gliders/dac/status/

ERDDAP Dataset Metadata & Imagery

DAC Metrics 649 Real-Time Glider Datasets Currently Available Deployments Glider Days Profiles

Individual Dataset Metrics Glider DAC Profiles GTS Observations Courtesy of OSMC: http: //osmc. noaa. gov/erddap/tabledap/index. html

Additional Tools Not Yet Implemented 1. JSON/CSV downloads of DAC and individual data set metadata a. Deployments, glider days and profiles i. Daily ii. Monthly iii. Yearly b. Dataset metadata c. Dataset geographic tracks 2. Dataset Metadata a. Global Attributes b. Variables i. ACDD/IOOS attributes ii. Standard names & units 3. Data Provider/Operator Suggestions on Additional Features?

Questions?