The Unified Access Framework UAF Philosophy progress and

The Unified Access Framework (UAF) Philosophy, progress, and plans DAARWG Meeting, Seattle, Nov. 2011 Kenneth. Casey@noaa. gov Julie. Bosch@noaa. gov Tina. Chang@noaa. gov Scott. Cross@noaa. gov Roy. Mendelssohn@noaa. gov Steven. C. Hankin@noaa. gov Jordan. Alpert@noaa. gov Jim. Sargent@noaa. gov Ted. Habermann@noaa. gov John. Relph@noaa. gov Bob. Simons@noaa. gov David. Neufeld@noaa. gov Upendra. Dadi@noaa. gov Rich Signell (rsignell@usgs. gov ) Phil. Cogbill@noaa. gov Glenn. Rutledge@noaa. gov Mike. Grogan@noaa. gov Jeff. Budai@noaa. gov Steve Hankin (PMEL), Kevin O’Brien (PMEL/JISAO), and the NOAA UAF team

Review … continuing into new material Nov. 2011 NOAA/UAF 2

GEO-IDE a strategy for enterprisewide integration of NOAA data (i. e. adopting standards and practices to achieve data interoperability) 2005 -06 66 pgs (by “DMIT”) Why is this a hard problem? Nov. 2011 NOAA/UAF 3

NOAA-world weather forecast (time critical) fisheries management (regulatory concerns) nautical charting climate, ocean, atmosphere research response and restoration … the list goes on … Nov. 2011 NOAA/UAF 4

Different disciplines have different concepts of ‘data’. Each develops solutions that make sense to them. Getting people (and organizations) to change habits is difficult! (and data management has often been an after thought. ) Nov. 2011 NOAA/UAF 5

The accepted approach: build a “system of systems” Wrap existing systems with loosely coupled, standardized services a Service Oriented Architecture GEO-IDE Con-ops outlines such a plan. How to build it with a largely volunteer team? Nov. 2011 NOAA/UAF 6

Tried and true approach … 1. 2. 3. 4. 5. Generate use cases Define requirements Write a Concept of Operations … and an Implementation Plan Assemble (volunteer) teams to implement NOT ! Nov. 2011 NOAA/UAF 7

An alternative (‘agile’) approach Don't Solve Problems -- Copy Success -- Nov. 2011 NOAA/UAF 8

Why ‘agile’ is attractive Because inevitably: funding is much smaller than needed; collaborations are more difficult than anticipated; infrastructure is being built on a background of rapidly evolving technology Nov. 2011 NOAA/UAF 9

Why ‘agile’ is attractive Change equals risk. Mitigate risk by following a strategy of incremental change that serves users (increasingly) well at every stage of evolution. infrastructure is being built on a background of rapidlya evolving technology “Visualize decade away. Build things that work today. ” Nov. 2011 NOAA/UAF 10

Agile Principles (condensed from the ‘Agile Manifesto’) Working software is the meaningful measure of progress: ensure early, continuous, frequent releases Emphasize simplicity - maximize the work not done Build projects around motivated individuals. Give them the support they need and entrust them The best architectures, requirements, and designs emerge from self-organizing teams Welcome changing requirements Scientists (‘business people’) and developers must work together on a daily basis Nov. 2011 NOAA/UAF 11

Lemma “Don’t let perfect be the enemy of good” Nov. 2011 NOAA/UAF 12

Projects: Data formats: (too many to name) net. CDF GRIB HDF … Service What “success” did UAF chose to copy? net. CDF-CF-DAP-THREDDS (WMS) stack: Year 1 focused on gridded datasets. Applications: Gr. ADS Users: Nov. 2011 Matlab Google Earth Arc. GIS IDV Ferret LAS ERDDAP (too many to name) NOAA/UAF … 13

Who is providing data this way? Modelers – AR 4&5, GFDL, NCAR, … Satellite programs – GHRSST, Path. Finder, Coast. Watch, … NCEP weather and ocean forecasts – GRIB files served via NOMADS Coastal (“HF”) radar A growing list of observations programs – Argo, Ocean. Sites, tide gauges, … Adoption by OGC is well underway Nov. 2011 NOAA/UAF 14

How to reach users? Through their preferred tools (without downloading files) Nov. 2011 NOAA/UAF 15

Model 1: UMASS-ECOM Model 2: UMAINE-POM Data: SST 2008 -Sep-08 07: 32 Desktop access in Matlab Nov. 2011 NOAA/UAF 16

Access in Arc. GIS using the Environmental Data Connector (EDC) Nov. 2011 NOAA/UAF 17

Desktop access in Ferret Nov. 2011 NOAA/UAF 18

Desktop access in Gr. ADS Nov. 2011 NOAA/UAF 19

UAF home page Instructions for end users: how to access data through their favorite applications Nov. 2011 NOAA/UAF is experimenting to see how documentation may be shared by other projects… 20

Under the hood Projects with data to provide – make their data available as net. CDF-CF (or other TDS-compatible format) – host a THREDDS / OPe. NDAP server Lets look at the tools we have to link projects together … Nov. 2011 NOAA/UAF 21

UAF “network topology” a tree defined in THREDDS (XML) GEO-IDE/UAF Nov. 2011 NAVO IOOS National Partners Coastwatch NGDC NOMADS NDBC NMFS NWS NESDIS PFEG GFDL ESRL PMEL AOML OCO OAR NOAA Affiliated NOAA/UAF IOOS Regional Partners AOOS CENCOOS SCCOOS NANOOS PACIOOS GLOS NERACOOS MACOORA SECOORA CARICOOS GCOOS NOAA 22

Not so fast, kiddo! compliance with CF conventions is inconsistent files commonly are not aggregated into logical datasets metadata are often in need of enhancement Perspective – this is not an unusual situation: Standards compliance problems are *the norm* Divergent dialects often pile up (e. g. GRIB, BUFR) UAF tools offer a solution … Nov. 2011 NOAA/UAF 23

‘Nc. ML’ can be used to repair problems (*) e. g. Improve CF compliance by adding “standard_name” attribute to GRIB data <variable name="vorticity"> <attribute name="standard_name" value="atmosphere_absolute_vorticity“ /> </variable> The file, itself, is untouched. The ‘virtual file’ seen through the services conforms to standards. (*) IOServiceprovider modules also important … not discussed here Nov. 2011 NOAA/UAF 24

Nc. ML for aggregation e. g. Aggregate three 1 -year files of the same (say) model run <aggregation type="join. Existing“ dim. Name="Time. Axis"> <netcdf location=“year 1. nc" ncoords=“ 365"/> <netcdf location=“year 2. nc" ncoords=“ 365"/> <netcdf location=“year 3. nc" ncoords=“ 365"/> </aggregation> A long time series ‘virtual file’ is seen through the services. Nov. 2011 NOAA/UAF 25

UAF Team members are helping data managers, person-to-person, to improve the data services from their projects (a gradual, but important process) in parallel … Nov. 2011 NOAA/UAF 26

UAF ‘RAW’ catalog IOOS National Partners NAVO NGDC NOMADS Coastwatch NDBC NMFS NWS NESDIS PFEG GFDL ESRL PMEL AOML OCO OAR NOAA Affiliated IOOS Regional Partners AOOS CENCOOS SCCOOS NANOOS PACIOOS GLOS NERACOOS MACOORA SECOORA CARICOOS GCOOS NOAA ‘RAW’ UAF ‘CLEAN’ catalog NOAA IOOS Regional Partners AOOS CENCOOS SCCOOS NANOOS PACIOOS GLOS NERACOOS MACOORA SECOORA CARICOOS GCOOS IOOS National Partners NAVO NGDC NOMADS Coastwatch NDBC NMFS NWS NESDIS PFEG GFDL ESRL PMEL AOML OCO OAR ‘CLEAN’ NOAA Affiliated Developing the UAF Catalog Cleaner (a ‘web crawler’) Nov. 2011 NOAA/UAF 27

The Catalog Cleaner 1. Crawl the raw catalog 2. Extracts metadata from the files, themselves, and from THREDDS into a relational database 3. Process the data base to detect aggregations, etc. 4. Create new THREDDS XML that is aggregated and metadata-cleaned Nov. 2011 NOAA/UAF 28

‘raw catalog’ snippit Nov. 2011 Optional documentation Services: Usually (beyond. OPe. NDAP. what is Often WMS. Sometimes Optionalinside the file) ‘ 09 Carbon Tracker more. viewers files from ESRL NOAA/UAF 29

‘clean catalog’ -- same data, but … augmented with • Uniform services • Uniform viewers • improved metadata Nov. 2011 uniform services uniform viewers NOAA/UAF 30

• Uniform services Simple interface to get a subset Nov. 2011 ISO-standard metadata quality metadata assessment NOAA/UAF 31

Nov. 2011 NOAA/UAF 32

Nov. 2011 NOAA/UAF 33

Godiva 2 uses the WMS map service (as do other GIS apps) Nov. 2011 NOAA/UAF 34

Google Earth Segue to desktop tools Sections and Hofmullers Propertyproperty plots Vector plots Animations Line plots Differencing Analyses Nov. 2011 NOAA/UAF 35

and ERDDAP provides … Nov. 2011 NOAA/UAF 36

. kml REST URL access to data subsets in several formats (accessible through home-grown scripting of many types) . nc . mat Nov. 2011 NOAA/UAF 37

Data Discovery ISO-standard metadata Nov. 2011 Crawl the clean catalog. Create ISO metadata NOAA/UAF 38

Agile principle: “Maximize the work not done” Who has already built reasonably mature discovery portals (preferably free) ? • Unidata’s RAMADDA • GI-CAT (able to crawl THREDDS catalog) • Geoportal (able to harvest ISO metadata) Nov. 2011 NOAA/UAF 39

Data Discovery using ESRI Geoportal NODC is leading the UAF Geoportal investigations

Data Discovery using ESRI Geoportal Enter Search terms

Data Discovery using ESRI Geoportal Search Results

Data Discovery using ESRI Geoportal Refined Search New Results

Data Discovery using ESRI Geoportal Expand Selection

Data Discovery using ESRI Geoportal Available Services

Data Discovery using ESRI Geoportal Available Services

Data Discovery using ESRI Geoportal View metadata details about the dataset

Data Discovery using ESRI Geoportal Available Services

Data Discovery using ESRI Geoportal Direct link into the THREDDS Data Server

Data Discovery using ESRI Geoportal Available Services

Data Discovery using ESRI Geoportal This seque to LAS demonstrates how Geo. Portal can be trained to recognize a uniform set of UAF viewers

Full disclosure: Geoportal is very powerful, but … • difficult to set up • immature user interface Nov. 2011 NOAA/UAF 52

In situ observations • are characterized by their ‘feature type’ (a time series, a track line, a vertical profile , …) • individual feature instances are degenerate (1 D) grids • the UAF framework just described already handles individual 1 D features • many data providers are creating such files (Argo, Ocean. Sites, …). Growth is robust. • but users want to interact with collections of features • standards and practices are in flux Nov. 2011 NOAA/UAF 53

Collections of in situ observations Candidates: 1. CF ‘Discrete Geometries’ • Chapter 9 of CF 1. 6, May 2011 • the ‘natural’ (? ) extension for UAF • but … • a new API yet to be released (TDS 4. 3) • Java support only • few tools available Nov. 2011 NOAA/UAF 54

Collections of in situ observations Candidates: 2. ‘DAPPER’ (OPe. NDAP Sequences) • solid technology • good prototype tools and examples already years ago • but … • poor standardization of semantics (conventions for lat, long, time, …) • flagging momentum … Nov. 2011 NOAA/UAF 55

Collections of in situ observations Candidates: 3. OGC Web Feature Service (WFS) • much momentum in the GIS community • but … • by design little standardization of semantics • communities must develop ‘application schemas’ • success rates poor, momentum flagging … Nov. 2011 NOAA/UAF 56

Collections of in situ observations Candidates: 4. OGC Sensor Observation Service (SOS) • significant momentum in the earth science community • but … • only the request protocol is standardized • many choices for semantics: “SWE common” , KML, custom ASCII/JSON/etc. • few tools, interoperability concerns Nov. 2011 NOAA/UAF 57

Collections of in situ observations What to do? • Agile philosophy: • Maximize the work not done • Don’t let perfect be the enemy of good • UAF isn’t resourced to undertake R&D ($ + risk) Nov. 2011 NOAA/UAF 58

Collections of in situ observations UAF approach: 1. Explore the potentials of ERDDAP. It provides • IOOS-compatible SOS • ‘Table. DAP’ and • commitment to support many protocols other R. E. S. T. outputs 2. Where data are in databases: install ERDDAP • e. g. OSMC (in situ ocean obs) 3. Prototype TDS 4. 3 / CF Discrete Geometries • Nov. 2011 Utilize from ERDDAP and LAS (Arc. GIS soon after? ) NOAA/UAF 59

Collections of in situ observations UAF approach: 4. Vigorously promote 1 D CF features • Nov. 2011 Here’s why: • Available today, understood by data providers, momentum exists • Compatible with UAF tools and practices • Provides immediate value to users NOAA/UAF 60

Collections of in situ observations UAF approach: 4. Vigorously promote 1 D CF features • Efforts will pay off (low risk) • Gets the data ‘out there’ (open, available) • Metadata-rich • Discoverable through standard frameworks • Future TDS versions will offer aggregation • Nov. 2011 miracles will happen for free: SOS, improved discovery, etc. NOAA/UAF 61

To increase data integration, what question should we be asking? Is it: “How do I build a system that will […]? ” Or is it: “How do we most rapidly narrow the gap between what we know technology is capable of delivering, and what our actual practices are today? ” What is the right balance of strategies: ‘Design and build’ vs. ‘Agile’ ? Nov. 2011 NOAA/UAF 62

Discussion Nov. 2011 NOAA/UAF 63