Interoperability What does it mean and how do

Interoperability What does it mean and how do we improve it? Jebb Q Stewart Chapel Hill, NC Thursday, July 11 th, 2013 NOAA Earth System Research Laboratory. Boulder , CO. Affiliated with Colorado State University Cooperative Institute for Research in the Atmosphere (CIRA)

Overview NOAA Earth Information System (NEIS) Climate. gov Data Interoperability Project Interoperability Challenges and Ideas to Improve Interoperability

NOAA Earth Information System (NEIS)

NEIS -The Concept NOAA Earth Information System (NEIS) is a framework of layered services designed to help NOAA’s mission areas by facilitating the discovery, access, integration, and under-standing of all NOAA (past, present, and future). Framework provides the capability to answer questions that require data from different data sources regardless of format or location.

Improving the User Experience How does industry make sense of massive amounts of information? • Applying existing concepts to NOAA data.

Standards Open Geospatial Consortium (OGC) WMS WFS WCS SOS WFS WCS CSW WFS – Web Feature Service WCS – Web Coverage Service SOS – Sensor Observation Service WMS – Web Mapping Service CSW – Catalog Service for the Web WPS – Web Processing Service

Searching and Metadata § Using Esri Geoportal (http: //geoportal. sourceforge. net/) as metadata repository and to harvest metadata from other sources. § Using Apache Solr (http: //lucene. apache. org/solr/) to provide additional functionality and improve discoverability. • Advanced Full-Text Search Capabilities: o Facets. o Full Document Storing (allows recreation). o Supports geospatial/temporal searching. • Optimized for High Volume Web Traffic § Solr instance indexes information from harvested metadata using Open Geospatial Consortium (OGC) Catalogue Service for the Web (CSW) standard, and directly from data access services.

Data Access § Currently support a variety of data formats: • KML • HTML • FIM native grids • Movies • ERDDAP (Environmental Research Division’s Data Access Program) • Open Geospatial Consortium (OGC) Web Mapping Service (WMS) § Built additional data access services to simplify requests for time sequences of rendered data making them available to NEIS/Terra. Viz. § Proxy and cache image requests to improve speed and reliability.

Visualization -- Terra. Viz • 3 D visualization tool for Earth datasets • Developed by NOAA, with full control of source code • Written in C#; similar to Java but runs natively ESRL Global Systems Division http: //esrl. noaa. gov/neis

About Unity is a commercial game engine that excels at rendering 3 D (and 2 D) scenes Develop once, then run on Windows, Mac, Linux, i. OS, Android, consoles, and browsers Millions of dollars in research and development Good community support and documentation, tutorials >800, 000 registered developers ESRL Global Systems Division http: //esrl. noaa. gov/neis

ESRL Global Systems Division http: //esrl. noaa. gov/neis

Partnerships § Climate. gov Data Interoperability Team § Groups o Open Geospatial Consortium (OGC) Standards Committees o NOAA Environmental Data Management Committee (EDMC) o NOAA Data Management Integration Team (DMIT) o NOAA Unified Access Framework (UAF) group o NSF Earth Cube o FIM/WRF Modeling o GIS Committee ESRL Global Systems Division

Climate. gov Data Interoperability Project

Vision “Interoperability is the ability for users to discover the available climate data, preview and interact with the data, and acquire the data in common digital formats through a simple web-based interface. ”

Prototyping for select Use Case • Built a proof of concept Data Interoperable Platform • The built system is “file format agnostic, ” meaning the pilot system will locate and display the data regardless of what format they’re archived • The web based client was developed using javascript libraries from Open. Layers and JQuery. Open. Layers library provides javascript utilities to interact with a variety of data and metadata services. JQuery provides utilities to construct the layout of the web page itself

How It Works • • The client allows users to interact with the various services without knowledge of how the services work. The client, and underlying libraries, handle formatting specifics for each service, managing requests against the service, and parsing the response for display to the user Using the search functionality, user’s can discover and access with the data above Once the data has been located, it can be loaded to preview a visualization of the data Once data has been selected, a user has further controls to filter underlying data based on time, keywords, or geographic location Once user has defined constraints, the user can extract this information and download the data to their local system for further analysis The codes are available to download and fork out in github: https: //github. com/Climate. Data/interoperability

CPC Data Served through WMS

Acknowledgements • • Sudhir Raj Shrestha (NOAA-CPC) Steve Ansari (NOAA-NCDC) Kevin O’Brien (NOAA-PMEL) David Herring (NOAA-CPO) Mark Phillips (UNCA) Micah Wengren (NOAA) Mike Halpert (NOAA-CPC)

Interoperability

Interoperability The ability to discover, access, view, interact, and integrate data regardless of format or physical location. Components of an Interoperability System • Format Agnostic. • Owner/Physical Location Agnostic. • Platform Agnostic. • Preview Capabilities. • Semantics/Ontology/Vocabulary. • Machine to Machine Communication.

Data Integration Physical Chemical Biological FIM Carbon Tracker Biological, Chemical, and Physical data are all interrelated

Interoperability Why do we want it? § Improve accessibility § Foster data exploration and use. § Decrease complexity § Provide framework for new tools and applications. § Possibilities are endless.

Interoperability is not It is not a web page or list of links like ftp • Not easily parsed or understood by a machine • No metadata It is not a service • Some optional parameters help interoperability • WMS max. Width/max. Height It is not a search engine • Found my data, how do I get it? What is this format? It is not anything that requires either human intervention or the machine to guess and try again

Driving Factors Whitehouse – Open Data Rules to Enhance Government Efficiency and Fuel Economic Growth • Order requires that, going forward, data generated by the government be made available in open, machine-readable formats, while appropriately safeguarding privacy, confidentiality, and security. http: //www. whitehouse. gov/sites/default/files/omb/memoranda/2013/m-1313. pdf http: //project-open-data. github. io/policy-memo/ NOAA Science Advisory Board (SAB) Environmental Information Services Working Group (EISWG) ‘Executive Summary’ • Recommends developing an Open Weather and Climate Services (WCS) in which both NOAA and the community share equal and full access to NOAA information and development http: //www. sab. noaa. gov/Doc/Towards-Open-Weather-and-Climate-Servicesreport-and-transmittal_12_23_11. pdf

Interoperability Readiness Levels • Measureable indicators… Credit: NASA ESDS Technology Infusion Working Group

Impacts Physical Framework built towards standards, not data. Chemical Biological Important Because: Ø NOAA data ready for action. Services model facilitates agile response to events. Services can be combined or reused quickly, upgraded or modified independently. Ø Any data available through framework can be operated on or combined with other data. Integrated standardized formats and access. Ø New and existing systems have access to wide variety of NOAA data. Any new data added, easy incorporated with minimal to no changes required. ESRL Global Systems Division

Challenges and Improving Interoperability

Finding and Discovering Data Challenges § Finding data often requires previous knowledge or understanding of what you are searching for § Having information readily available on data and services for searching. Our environment evolves rapidly § Harvesting versus Aggregated searching § Missing or Incomplete Data § Derived or generated products/data not well documented or discoverable

Finding and Discovering Data Recommendations Metadata must contain information to allow machine to machine communication, such as: Service Information Need a Service End. Point Use ISO SV_Service. Identification Time, periodicity (daily, monthly, yearly) Use EX_Temporal. Extent with ISO 8601 for Time and Duration Units Use ISO 19115 Proposed Revision MD Sample. Dimension Previews ISO 19115 MD Browse. Graphic Styling, Ontology/Semantics Next Slides § Automated tools to generate Metadata o nc. ISO § Investigate how to document “possible” derived or generated data based on process.

Data Access and Display Challenges § Lack of adherence to specifications and consistency behind standards. § Incomplete or Work in Progress Standards. § Generation of graphical representations of data. How do I stylize point information? What color palette is the best for this data? § Server uptime and data availability. Metadata records say data is available for time but server may be down during access. § Derived or generated data, what process to use and how to use it?

Data Access and Display Recommendations § Metadata uniformity helps improve user experience – need best practices, examples, and/or automated ways to produce information. o OGC Met. Ocean DWG and Others § Dashboard/Automated Testing Systems to give feedback to data providers on outages, adherence to standards, and metadata matchup to capabilities. o Rubrics, nc. ISO, etc… § Need information on how data should be displayed, imaged, stylized. Open Geospatial Consortium (OGC) Styled Layer Descriptor (SLD) and Symbology Encoding (SE). o Use ISO Online. Resource to point to style information.

Metadata Challenges Tags and Keywords § § Metadata keywords inconsistent. Lack of meaningful tags. Keyword repetition – same keywords are used on several data sets. Semantics, Hierarchy, Taxonomy, Ontology, Relationships – All difficult to infer. § Missing Information Examples: “Sea Surface Temperature” “SST” “Sea” “Surface” “Temperature” “NOAA” vs. “National Oceanic Atmospheric Administration”, “Wind” and “Winds” Ocean -> Ocean Circulation -> Ocean Currents

Metadata Recommendations § Be Aware of existing ontologies and keywords o GCMD Keywords: http: //gcmd. nasa. gov/Resources/valids/archives/keyword_list. html o NASA JPL SWEET Ontology http: //sweet. jpl. nasa. gov/ § Allow user defined keywords (Crowdsourcing) § To what extent can we auto generate this information o nc. ISO, etc… § Metadata validation o Rubrics, Dashboards, etc…

Communication Stay Involved and Connected: Organization Groups Within NOAA: Environmental Data Management Committee (EDM), Data Management and Integration Team (DMIT) OGC Committees (Like OGC Met. Ocean DWG) ESIP NSF AGU Others? Wiki’s: NOAA EDM -- https: //geo-ide. noaa. gov/wiki/ ESIP Others? Prototypes or Proof of concepts

Discussion

Other Activities

NEIS Development Activities Discovery Improve discoverability and ease of finding data through: § Continued ontology development by linking and improving ontologies. § Improved analytics. Learn what is being used and when to provide better results for future searching. § Creating profiles for environmental awareness. § Providing capability to weight results depending on context.

NEIS Development Activities ‘Big Data’ NOAA’s Big Data is different. § NPP data (~ 4 TB / day and growing) § GOES-R § New global forecast models, rapidly increasing in size. § Vast amounts of historical data Data are ever increasing in size. Improve accessibility to ‘Big Data’ through: § Providing tools allowing seamless integration of data across time and space regardless of data size. § Minimize data we transfer and avoid data duplication. § Move processing close to data. § Allow users to collaborate on said data.

And Beyond Continue development, integrating and leveraging new and emerging technologies to meet NEIS goal ‘any data, any location, any platform, now’ § Perform processing within cloud environment and with high speed connectivity to data sources, taking advantage of large processing power within cloud infrastructure. § Send graphics and server side processed/rendered/streamed data to GUI, improving bandwidth utilization. § Take advantage of fast networking to make remote requests and processing appear like local application. § Similar to how the concept of the Amazon Silk Browser. ESRL Global Systems Division

Questions? Jebb. Q. Stewart@noaa. gov http: //www. esrl. noaa. gov/neis