DIVER Data Integration Visualization Exploration and Reporting An

DIVER (Data Integration, Visualization, Exploration, and Reporting): An Introduction December 2, 2020 Nicolas Eckhardt NOAA - NOS - Office of Response & Restoration Assessment & Restoration Division: Spatial Data Branch 1

Overview DIVER (Data Integration, Visualization, Exploration, and Reporting) DIVER is a data warehouse, query tool, and collaboration application. The DIVER approach integrates standardized datasets so users can query across data holdings and download information and results. 2

Shoreline Data Marsh Assessment Oyster Collections Telemetry Data DWH Damage Assessment Data Toxicity Data • 20, 000 trips for field data collection • 1 million field data forms and Water Column related electronic files • 100, 000 water, tissue, oil and sediment samples Seafood Safety • 15 million+ database records • 30 terabytes of data • Data Referenced in many Marine Mammal & Publications/Journals Turtle Assessment 3

What is DIVER? Processing Collates, standardizes and transforms source data Data Access Data Warehouse Stores and serves integrated data • • • Query / Download Visualization Reporting / Analytics Public access / Sharing Publications (Papers) Collaboration 4

DIVER: Key Capabilities Data Integration and Sharing • Integrate and standardize data from multiple sources (e. g. field-collected data, laboratory data, monitoring data, analysis) • Query, Export, Reporting tools • Federal data sharing requirements (e. g. Open Data Policy; PARR; ISO Metadata) Cloud-based Infrastructure Secure • Federal IT Security Requirements Scalable/Flexible • Evolving application to meet evolving needs 5

DIVER Security (Access Levels) • Access is granted by Region/Activity/Workspace. • All accounts are approved by ORR staff – Check with Federal/State partners for account permissions • Users restricted to seeing only what their permissions allow. • Trustee only, Trustee and RP options 6

Common Data Models (standards) Core fields Data type specific models Samples: Chemistry, biological+ Bioassay: Toxicity testing and results Bioassay Field Observations and Measurements: Field Observations/ Measurements Ocean Data Telemetry Photographs Projects shoreline, marsh, birds and mammals; biological data Oceanographic: Cruise-collected sensor data Telemetry: Whales, dolphins, turtles, tuna. . Photography: Geolocation, Keywords Restoration data: Project tracking data SCAT (Shoreline Cleanup Assessment 7 Techniques): In Development

DIVER Data Specification Document • Public DIVER - Data Overview Section –https: //www. diver. orr. noaa. gov/web/guest/dataoverview –Describes underlying data structure –Details field information and Valid Values –Data exchange methods –Tables in Appendices describe fields 8

FIELD FORMS, DATA TEMPLATES, AND GUIDELINES • Public DIVER - Data Overview Section • https: //www. diver. orr. noaa. gov/web/guest/fieldforms-and-templates • Field & Laboratory protocols • Data templates • Chain of Custody • Field data collection forms • Help standardize data collection and intake 9

Data Integration Protocols and Standards Field Forms (Data) Transcribe Electronic Data 10

Data Integration Unstructured Data Structured Data Metadata Samples Metadata Core fields Bioassay Digital Data Form Metadata Field Observations/ Measurements Ocean Data Metadata Telemetry Photographs . CSV Projects 11

Public DIVER website Public website for Assessment & Restoration data https: //www. diver. orr. noaa. gov 12

National DIVER Portal Overview 13

National DIVER Portal Overview Login environment; data sharing status Help documents, data details and data management tools including OR&R Activities Database Quick Access to query data by Regions The Portal is organized into Regions Activities and Workspaces. Regional pages provide DIVER Explorer filtered to regional data and case-specific data based on user access. Workspaces can contain File Collections, collaboration and data entry tools. https: //portal. diver. orr. noaa. gov Quick Access to ERMA, online mapping and visualization tool 14

DIVER WORKSPACE ORGANIZATION • Regions are organized in a hierarchical order, with five regions (Great Lakes, Northeast, Northwest & Arctic, Southeast, and Southwest & Pacific Islands). • Activities are regional events that may include hazardous waste sites, oil spills, drills, restoration projects, and special activities • Workspaces are where data management and collaboration occur. (Next Slide) 15

DIVER Workspaces Key Development Functionality: Data Managers are able to configure and customize Activities and Workspaces for case-specific data needs. 16

Activities • Activities are events such as cases, spills, workshops or drills • Activities contain Workspaces which can contain modules for managing data, documents, collaboration or querying data (Explorer). 17

Workspaces are for collaboration of information and data, data management and data exchange • • File Collections Dashboards Collaboration Tools Information about Data (Plans/Reports) Privileges are restricted to selected set of users (e. g. Trustees Only) 18 Workspaces can be regional or activity constrained

Workspace: File Collections Data Upload: Structured (spreadsheets) and unstructured (scanned PDF, word docs) 19

Workspace: Document Libraries 20

Dashboard 21

DIVER Explorer: Query Result DIVER Explorer is a query tool. Filter, map and download. Dashboard display. 22

DIVER Explorer Guided Queries and Build New Query Project Queries Environmental Data Queries Explore Environmental Data 23

DIVER Explorer Query Filters Fields Values Selections Filter 24

DIVER: Data Access 25

DIVER Explorer: Export Packages Mapping/GIS Metadata & Study Notes Data Packages Expanded Data Services 26

Dashboards 27

Restoration Data Project details, activities & status Locations Monitoring • $8. 8 billion settlement payed out over 15 years • Funds allocated to 7 Trustee Implementation Groups (TIGs) across 15 resource types • Projects will be implemented by 17 trustee agencies • Commitment to data-driven adaptive management Budget, receipts & expenditures Environmental compliance Accomplishments 28

DIVER Portal: Restoration Technical Implementation Groups (TIGs) have customized content and group specific permissions 29

Gulf Spill Restoration Website (Public) 30

Questions? 31