Status Report on NASAs Earth Observing Data and
Status Report on NASA’s Earth Observing Data and Information System (EOSDIS) Dawn Lowe / Andrew Mitchell September 2016
Earth Science Data Systems Mission Statement The Earth Science Data System Program oversees the lifecycle of earth science data with the principle goal of maximizing the scientific return from NASA’s missions and experiments for research and applied scientists, decision makers and society at large. 2
Topics EOSDIS Overview Metrics Status of NASA SNPP Data Production Future Missions Update on EOSDIS Cloud Technology Prototypes 3
Data Sources Type Satellite/on-orbit Missions Airborne missions In Situ Measurement missions Example Missions Terra, Aqua, Aura, Suomi-NPP, SORCE, GPM, GRACE, Cloud. Sat, CALIPSO, etc. Ice. Bridge, Earth Ventures (5+ missions), UAVSAR, etc. Field campaigns on land (e. g. , LBA-ECO) and in the ocean (e. g. , SPURS) Applications support Near-real time creation and distribution of selected products for applications communities Earth Science Research support Research products from efforts like MEa. SUREs. This also includes data from older, heritage missions (prior to EOS Program) that the DAACs rescue – e. g. , Nimbus, Sea. Sat 4
National Aeronautics and Space Administration Science Investigator-led Processing Systems Measurements of Pollution in the Troposphere (MOPITT) Visible Infrared Imaging Radiometer Suite (VIIRS) Atmosphere Ozone Monitoring Instrument (OMI) Microwave Limb Sounder (MLS) Tropospheric Emission Spectrometer (TES) MODIS Adaptive Processing System (MODAPS) Advanced Technology Microwave Sounder (ATMS) & Cross-track Infrared Sounder (Cr. IS) Sounder Ocean Data Processing System (OCDPS) Visible Infrared Imaging Radiometer Suite (VIIRS) Land Visible Infrared Imaging Radiometer Ozone Mapping Suite (VIIRS) Profiler Suite Ocean (OMPS) Ozone Advanced Microwave Scanning Radiometer for EOS 2 (AMSR-E/2) SIPSs perform forward processing of standard products, and reprocess data to incorporate algorithm improvements. www. nasa. gov 5
National Aeronautics and Space Administration Discipline-oriented Data Centers ASF SDC SAR Products, Sea Ice, Polar Processes, Geophysics SEDAC Human Interactions, Land Use, Environmental Sustainability, Geospatial Data LP DAAC Surface Reflectance, Land Cover, Vegetation Indices GES DISC Global Precipitation, Solar Irradiance, Atmospheric Composition and Dynamics, Global Modeling NSIDC DAAC Snow and Ice, Cryosphere, Climate Interactions, Sea Ice PO. DAAC CDDIS Gravity, Sea Surface Temperature, Ocean Winds, Topography, Circulation & Currents Space Geodesy, Solid Earth La. RC ASDC GHRC DAAC Hydrologic Cycle, Severe Weather Interactions, Lightning, Atmospheric Convection Radiation Budget, Clouds, Aerosols, Tropospheric Chemistry OBPG Ocean Biology, Sea Surface Temperature MODAPS/ LAADS MODIS Level-1 and Atmosphere Data Products ORNL DAAC Biogeochemical Dynamics, Ecological Data, Environmental Processes www. nasa. gov 6
Role of EOSDIS DAACs were selected and established based on the Earth Science discipline expertise and heritage of their host organizations. They provide unique support and expert services to their user communities. They also provide data and services to the research community for comprehensive, cross-discipline studies needed to understand Earth as an interrelated system. DAAC Functions • • Ingest Level 0 data from EDOS Ingest higher level products produced by SIPS Perform processing of higher level products in some cases Archive assigned data sets Export metadata to the Common Metadata Repository (CMR) Provide user interfaces, tools, and services Distribute data to users (primarily electronically) Provide metrics data to ESDIS Metrics System (EMS) DAACs ensure safe stewardship of NASA’s data 7
DAAC Context Acquisition, e. g. , EOS Data Operations System (EDOS) NASA Sentinel Gateway Science Researchers Processing NOAA NCEP Precipitation Precise Orbit Determination ata e. D Educational Users Common Metadata Repository (CMR) cs el ri et ut O t pu Land Data Assimilation Systems od Global Modeling and Assimilation EOSDIS Core System / Discipline specific systems Applications Users M Models, e. g. , y illar Anc ata D Ancillary, e. g. , Me tad ata Science Computing Facilities (SCFs) Distributed Active Archive Center Sci enc Mission Processing Systems Raw Data & ta Da cts ce du ien Pro Sc ta Da Science Investigatorled Processing Systems (SIPSs) M EOS Metrics System (EMS) 8
EOSDIS-wide Elements and Capabilities Earth. Data: the EOSDIS website (Earthdata. nasa. gov) provides a focal point for cross-DAAC and EOSDIS content and news sharing, and access to Earth science data and services Common Metadata Repository (CMR): metadata catalog of NASA's EOS data and related data services (e. g. reformatting, pattern recognition) as well as Collection-level metadata for ~ 25, 000 data sets from broad international community. Essential underlying component for data discovery and access. Earth. Data Search Client (EDSC): easy-to-use access to EOSDIS services for Earth science data discovery, filtering, visualization, allowing users to download and access Earth observation data Near Real-Time Capabilities: Provided by “LANCE” (Land Atmosphere Near real-time Capability for EOS) which produces products within < 3 hours of observation. Near real-time capabilities are co-located with the standard science production facilities. Global Image Browse System / World. View: GIBS provides access to full resolution imagery derived from NASA products; Worldview Client allows users to explore GIBS imagery in a Google. Maps like manner. ESDIS Metrics System (EMS): collects and reports on data ingest, production, archive, and distribution across all EOSDIS data centers Earthdata Log-in (User Registration System): provides a centralized and simplified mechanism for user registration and account management for all EOSDIS system components. 9
15+ years of Earth Science Data Total EOSDIS Accumulated Data Archive Volume (Petabytes) Addition of AQUARIUS and ICEBRIDGE products Landsat-7 Data Migrated from LPDAAC to USGS Addition of ALOS PALSAR & UAVSAR products Addition of SNPP CERES&OMPS Deletion of MODIS Collection 4 Products TERRA AQUA ICESat AURA MEASURES 2006 MEASURES 2012 10
EOSDIS Products Delivered: FY 00 thru Aug 2016 121. 7 M Year Pre-decisional - For NASA Internal Use Only 11
FY 2016 EOSDIS Data Products Distributed by Country 12
SNPP Land Products for EOS Continuity Product Name MODIS VIIRS (S(EOS NPP, JPSS Terra and 1 and JPSS Aqua) 2) Planned Time Frame to the DAACs Land Surface Reflectance X X October 2016 MAIAC Product Suite * X X December 2016 BRDF/Albedo, NBAR X X November 2016 Land Surface Temperature X X December 2016 Vegetation Indices (VI) X X November 2016 FPAR X X November 2016 Fire and Thermal Anomalies X X October 2016 Burned Area X X March 2017 Snow Cover X X October 2016 * Includes surface reflectance, BRDF, snow fraction and aerosol retrievals over Land 13
SNPP Land Products for EOS Continuity (cont. ) Product Name MODIS (EOS Terra and Aqua) VIIRS (S-NPP, JPSS 1 and JPSS 2 Planned Time Frame to the DAACs Sea Ice Cover X X October 2016 Ice Surface Temperature - X November 2016 Land Cover Dynamics (Land Surface Phenology) X X April 2017 Note: Products are available in both instrument swath (Level 2) and gridded (Level 3), except MAIAC suite (Level 3 only). 14
SNPP Ocean Products for EOS Continuity Product Name MODIS (EOS Terra and Aqua) VIIRS (SNPP, JPSS 1 and JPSS 2) Planned Time Frame to the DAACs Aerosol Angstrom Exponent X X available now Aerosol Optical Thickness X X available now Chlorophyll a Concentration X X available now Diffuse attenuation at 490 nm X X available now Photosynthetically Available Radiation X X available now Particulate Inorganic Carbon X X available now Remote Sensing Reflectance X X available now Inherent Optical Properties (total and component absorption and scattering coefficients) X X available now Sea Surface Temperature X X available now Note: VIIRS/SNPP products are now available as six minute Level-1 and Level-2 files (vs. NOAA’s 86 second versions) and global 4 km produced using NASA algorithms and EDOS-provided raw data files. 15
VIIRS Level 1 and Atmosphere Products for EOS Continuity Product Name MODIS (EOS Terra and Aqua) VIIRS (SNPP, JPSS 1 and JPSS 2) Planned Time Frame to the DAACs Level 1 A X December 2016 Geolocation Fields X X December 2016 Level 1 B (Calibrated Radiances) X X December 2016 Cloud Mask X X January 2017 Aerosol Dark Target X X January 2017 Aerosol Deep Blue X X December 2016 Total Precipitable Water Vapor X X January 2017 Cloud Properties (Optical & Infrared) X X January 2017 Cirrus Reflectance X X December 2016 Joint Atmosphere Product* X TBD * Selected parameters from Level 2 products 16
SNPP Ozone Products for EOS Continuity EOS Aura OMI SNPP OMPS Nadir + Limb JPSS 1 OMPS Nadir JPSS 2 OMPS Nadir + Limb Planned Time Frame to the DAACs (OMPS-NPP Products) Calibrated, Geolocated L 1 B X X September 2016 (nadir) October 2016 (limb) Ozone nadir vertical profiles X X September 2016 Ozone total column X X September 2016 Aerosol Index X X Included in Ozone total column product High resolution vertical Ozone profiles - X January 2017 High resolution aerosol vertical profiles - X November 2016 Product Name 17
Sounder Products for EOS Continuity Product Name AMSUA/AIRS (EOS Aqua) ATMS/Cr. IS (S- Planned Time Frame NPP, JPSS 1 to the DAACs and JPSS 2) MW Level 1 B radiances X X January 2017 IR Level 1 B radiances (reduced resolution) X X January 2017 IR Level 1 B radiances (full resolution) X X January 2018 Level 2 Products* (reduced resolution) X X October 2017 (beta) Level 3 Gridded products* (reduced resolution) X X February 2018 Level 2 Products* (full resolution) X X October 2018 Level 3 Gridded products* (full resolution) X X February 2018 * Atmospheric Temperature & Water Vapor profiles, Surface Temperature and Emissivity, Total Cloud Liquid Water, Cloud Properties (fractional cover, cloud top temperature, cloud top height), Ozone, Carbon Monoxide, Methane, Cloud Cleared IR Radiances 18
CERES Products for EOS Continuity CERES (EOS Terra and Aqua) CERES (SNPP, JPSS 1 and JPSS 2) Planned Time Frame to the DAACs Bi-Directional Scan (BDS) Radiance X X Available now ERBE-like Instantaneous TOA Estimate (ES-8) X X Available now ERBE-like Monthly Regional Averages (ES-9) X X Available now ERBE-like Monthly Geographical Averages (ES-4) X X Available now Single Scanner Footprint TOA/Surface Fluxes and Clouds (SSF) X X Available now Hourly Gridded TOA/Surface Fluxes and Clouds (SSF 1 deg. Hour) X X Available now Product Name
CERES Products for EOS Continuity (cont’d) Product Name CERES (SPlanned (EOS Terra NPP, JPSS Time Frame and Aqua) 1 and JPSS to the DAACs 2) Monthly and Daily Gridded TOA/Surface Fluxes and Clouds (SSF 1 deg-Month/Day) X X Monthly and Daily Zonal and Global Radiative Fluxes and Clouds (SYN 1 deg-Month/Day) X X Fall 2016 Cloud Type Histogram X X Summer 2016 Available now
ESDIS EOSDIS Future Mission Support CY 2016 2017 2018 2019 2020 2021 2023 2022 EOSDIS 11/11/16 5/10/17 SAGE-III ISS/ASDC 11/11/16 5/10/17 LIS on ISS/ GHRC CYGNSS/JPL-PO OCO-3 ISS/GESDISC 11/21/16 GRACE FO/JPL-PO TSIS-1 ISS/GESDISC ICESat-2/NSIDC ECOSTRESS ISS/LPDAAC GEDI LP & ORNL SWOT/JPL-PO Planned Data Release 5/20/17 1/1/17 JPSS-1 Launch 6/30/17 3/1/17 8/28/17 8/1/17 1/28/18 9/1/17 12/1/17 1/1/18 2/28/18 5/30/18 6/30/18 3/1/19 8/29/19 4/1/21 9/8/21 TEMPO/ASDC 12/1/21 NISAR/AST 12/30/21 PACE/OBDAAC 5/30/22 6/28/22 8/1/22 1/28/23 21
EOSDIS Cloud Prototypes Web Object Storage and distribution prototype • Goals: • Store and distribute data from Amazon S 3 • Capture performance and usage metrics to understand cost drivers for storage and egress fees (tall poles). • Work with Amazon on cost models to support petabyte scale archives. • Status: • ASF WOS is storing data and serving to the public using AWS S 3. • Migration of Ingest and storage software is underway. Ingest/Archive Prototype • Goal: • Develop software to utilize capabilities of commercial clouds for the efficient ingest, archive and management for the unique requirements for satellite, airborne and in situ earth science data. • Status: • Phase 1 of prototype demonstrated to DAACs in August • Phase 2 (additional data sets) will begin in October.
EOSDIS Cloud Prototypes Hosting Applications in the cloud offers the clearest path to improve efficiency. Over the next 2 years, core applications will be adapted to work within commercial cloud environments. • Metadata Search and Order System • Common Metadata Repository (CMR): Provides sub-second search and discovery services for over 250 M spatial objects using openly available APIs. Catalog of data from all DAACs and many international agencies. • Status: Cloud Migration has begun with creation of the database; anticipate completion in ~ 9 months • Visualization System • Global Imagery Browse Services (GIBS): Full resolution imagery from NASA instruments publically provided through open standards based APIs and the Worldview web client. • Status: Re-engineered GIBS workflow to replicate functionality in the Cloud. Demonstrated with VIIRS imagery. Additional data sets being added. • Earthdata Web Infrastructure • Platform for developing and hosting science websites and portals interacting with spatial data, includes requirement management, code development, testing and deployment to operations. • Status: Developers’ Portal section has been deployed to a Cloud, but is still under development. • Earthdata Login • Centralized mechanism for user self-registration and profile management across EOSDIS • Status: Migration to Cloud put on hold while Earthdata login approach is being reexamined.
Acronym List A&A AAAS AAG ACRIM ACSI ADEOS AETD AIRS ALOS AMASS AMSR ANGe API ARRA ASCI ASDC ASF ASTER GDS ATMS ATO BGP BMGT Bo. E CARS CDDIS CEOS CERES CF CLASS CM CMR CNES CO COFUR COMET COTS Cr. LS CSOC CSW CWIC CYGNSS C 1 DAAC DAP DAS DEM DISC DOE Authorization and Accreditation American Association for Advancement of Science Association of American Geographers Active Cavity Radiometer Irradiance Monitor American Customer Satisfaction Index Advanced Earth Observing System (Japan) Applied Engineering and Technology Directorate Atmospheric Infrared Sounder Advanced Land Observation Satellite Archival Management and Storage System American Meteorological Society Advanced Microwave Scanning Radiometer Archive Next Generation Application Programming Interface American Recovery and Reinvestment Act American Survey Customer Index Atmospheric Sciences Data Center (Langley) Alaska SAR Facility Advanced Spaceborne Thermal Emission and Reflection Radiometer Ground Data System Advanced Technology Microwave Sounder Authorization to Operate Border Gateway Protocol Bulk Metadata Generation Tool Basis of Estimate Climate Analysis Research Systems Crustal Dynamics Data Information System Committee on Earth Observations Satellites Cloud’s and Earth’s Radiant Energy System Climate and Forecasts Communication Link Analysis Simulation System Comprehensive Large Array-data Stewardship System Configuration Management Common Metadata Repository Centre National D’etudes Spatiales (France) Carbon Monoxide Cost of Filling User Request COnfiguration Management EOS Tool Commercial Off the Shelf Cross-Track Infrared Sounder Consolidated Space Operations Center Catalog Services for the Web CEOS WGISS Integrated Catalog Cyclone Global Navigation Satellite System Configuration 1 Distributed Active Archive Center Delivered Algorithm Package Data Assimilation System Digital Elevation Model Data and Information Services Center Department of Energy DR DRN DSCOVR Ebnet ECC ECHO ECS EDC EDG EDOS EDR EED EDSC EGS ELV EMD EMOS EMS EOC EOSDIS EPGN EPGS ERG ESDIS ESDSWG ESDT ESE ESIP ESISS ESRI ESSAAC ETE ETS EWOC FDS FMU FOS FOR FTP G&A GCMD GDAAC GDS GDEM GEOS GES GHRC GIBS Discrepancy Report Data Ready Notification Deep Space Climate Observatory EOS Backbone Network Earthdata Code Collaborative EOSDIS Clearing House EOSDIS Core System EROS Data Center EOS Data Gateway EOS Data & Operations System Environmental Data Records EOSDIS Evolution and Development Earth Data Search Client EOS Ground System Expendable Launch Vehicle EOSDIS Core System Maintenance and Development EOS Mission Operation System EOSDIS Metrics System EOS Operations Center Earth Observing System EOS Data & Information System EOS Polar Ground Network EOS Polar Ground Station EOSDIS Review Group Earth Science Data & Information System Earth Science Data Systems Working Group Earth Science Data Types Earth Science Enterprise Earth Science Information Partner ESA Space Information Systems Environmental Systems Research Institute Earth System Science and Applications Advisory Committee End to End EOS Test System ECHO Web Service Definition Language Ordering Component Flight Dynamics System Formatter/Multiplexor Unit Flight Operations Segment Flight Operation Review File Transfer Protocol General and Administrative Global Change Master Directory Goddard DAAC Ground Data System Global Digital Elevation Map Goddard Earth Observing System Goddard Earth Sciences Global Hydrology Research Center Global Imagery Browse Services 24
Acronym List GIM GLAS GMAO GN GOBBS GOE GPM GPP GPRA GRACE FO GRAVITE Geomatics Info. Magazine Geoscience Laser Altimeter System Global Modeling and Assimilation Office Ground Network Goddard Opportunities Bulletin Board System Ground Operations Exercise Global Precipitation Measurement Gross Primary Production Government Performance Results Act Gravity Recovery and Climate Experiment-Follow-On Government Resource for Algorithm Verification, Independent Testing, and Evaluation GSFC Goddard Space Flight Center GSIF Ground Station Interface Facility HDF Hierarchical Data Format HEG Hierarchical Data Format - Earth Observing System-Geographic Tagged Image File Format HIRDLS High-Resolution Dynamics Limb Sounder ICD Interface Control Document ICE Integrated Communication Effort ICESat Ice Cloud and Land Elevation Satellite IDN International Directory Network IDPS Interface Data Processing Segment IPR Initial Project Review IPY International Polar Year IRD Interface Requirements Document ISD Information System Division ISO Internal Organization for Standardization ISS-Rapid. Scat International Space Station Rapid Scat-terometer IST Instrument Support Terminal IT Instrument Team IT Information Technology IV&V Independent Verification & Validation I&T Integration & Test JPIP Joint Project Implementation Plan JPL Jet Propulsion Laboratory JPSS Joint Polar Satellite System JSC Johnson Space Center KNMI Koninklijk Nederlands Meteorologisch Instituut L-7 Landsat-7 - Land Remote Sensing Satellite LAADS Level 1 and Atmosphere Archive and Distribution System LAI Leaf Area Index LANCE Land Atmosphere Near real time Capability for EOS La. RC Langley Research Center La. TIS La. RC DAAC TRMM Information System LBA Large-Scale Biosphere Atmosphere Experiment in Amazonia LDB Long Duration Balloon LEOP Launch and Early Orbit Phase LP LPDC LRD LTO MD 5 MEa. SUREs METI MISR MLS MODAPS MODIS MOPFAS MOPITT MOR MOSS MPS MRR MSS NASDA NCAR NCCS NCDC NCR NCSA NEAT NST 4 NICSE NISN NIVR NOAA NPP NRC NRTPS NSOF NSIDC OB. DAAC OBPG OCDPS OCO ODPS OECD OES OGC OHCM Land Processes (DAAC) Land Processes Data Center Launch Readiness Date Linear Tape-Open Message-Digest algorithm Making Earth Science Records for Use in Research Environments Ministry of Economy, Trade and Industry Multi-Angle Imaging Spectro-Radiometer Microwave Limb Sounder MODIS Adaptive Processing System Moderate Resolution Imaging Spectro-Radiometer MOPITT operational fast forward model Measurements of Pollution in the Troposphere Mission Operations Review Mission Operations and Science System Mission Planning System Mission Readiness Review Management Subsystem National Space Development Agency (Japan) National Center for Atmospheric Research NASA Center for Climate Simulation National Climatic Data Center Nonconformance Report National Center for Supercomputer Applications Network Evolution and Architecture Transformation NPP Compatibility Test 4 NPP Instrument Calibration and Science Element NASA Integrated Support Network Netherlands Agency for Aerospace Programs New Obligation Authority National Oceanic and Atmospheric Administration Net Primary Production NPOESS Preparatory Project National Research Council Near Real-Time Processing System National Satellite Operations Facility NASA Science Internet National Snow and Ice Data Center Ocean Biology Distributed Active Archive Center Ocean Biology Processing Group Ocean Color Data Processing System Orbiting Carbon Observatory OMI Dutch Processing System Organization for Economic Co-Operation and Development Office of Earth Science (Code Y ) Open Geospatial Consortium Office of Human Capital Management 25
Acronym List OMAERO OMBRO OMCLDRR OMHCHO OMIDAPS OMOCLO OMPS OMTO 3 ORNL ORR PAD PALSAR PEATE PDR PDS PGEs PODAAC POP PPBE RAID RCN RDS REASon RF RFA RFP RIP ROSES RRR S 4 PA S 4 PM SAGE III SAN SAR SD 3 E SDPS SDS SEB SEDAC SI&T SIPS SLOC SMAP OMI standard product (Aerosol product using the multi-wavelength algorithm) OMI standard product (Bromine Oxide) OMI standard product (Cloud Pressure and Fraction-Raman Scattering) OMI standard product (Formaldehyde) Ozone Monitoring Instrument Data Processing System OMI standard product (Chlorine Oxide) Ozone Mapping and Profiler Suite OMI standard product (Total Column Ozone) Oak Ridge National Laboratory (DAAC) Operations Readiness Review Precision Attitude Determination Phased Array L-band SAR Product Evaluation and Analysis Tool Element Preliminary Design Review Production Data Set Product Generation Executives Physical Oceanography Distributed Active Archive Center Period of Performance Planning, Programming, Budgeting, and Execution Redundant Array of Inexpensive Disks Receipt Confirmation Notification Remote Data Storage Research, Education, and Applications Solutions Network Radio Frequency Request for Action Request for Proposal Retained Intermediate Product Research Opportunities in Space and Earth Sciences Release Readiness Review Simple, Scalable, Script-Based, Science Processing Archive Single Scalable Script-based Science Processor-mission Stratospheric Aerosols and Gas Experiment Storage Area Network Synthetic Aperture Radar Science Data Distribution and Depository Science Data Processing Segment Science Data Segment Source Evaluation Board Socio-Economic Data and Applications Center (CIESIN) System Integration and Test Science Investigator-led Processing Systems Single Lines of Code Safety and Mission Assurance Soil Moisture Active Passive SNF SNPP SOMO SORCE SORR SOW SPo. RT SRB SRR SSDO SSI&T SSR SST S/W SIMBIOS TADS TB TC IDEA TDRSS TES TIM TOTS TLM TRMM TRR TSDIS UPN UPS URS USGS VATC VIIRS WDS WGISS WIST WFF WSDL YTD Superior National Forest Suomi National Polar-orbiting Partnership Space Operations Management Office Solar Radiation Climate Experiment Segment Operational Readiness Review Statement of Work Short-term Prediction & Research Transition system at MSFC Storage Resource Broker System Requirements Review Science System Development Office Science S/W Integration and Test Solid State Recorder Sea Surface Temperature Software Sensor Intercomparison and Merger for Biological and Interdisciplinary Oceanic Studies Terra and Ancillary Data Support Terabyte Tropical Cyclone Integrated Data Exchange and Analysis System Tracking and Data Relay Satellite System Tropospheric Emission Spectrometer Technical Interchange Meeting Transportable Orbital Tracking System Telemetry (subsystem) Tropical Rainfall Measuring Mission Test Readiness Review TRMM Science Data & Information System Unique Project Identifier Uninterruptible Power Supply User Registration System United States Geological Survey Verification and Acceptance Test Center Visible Infrared Imager Radiometer Suite World Data System Working Group on Information Systems and Services Warehouse Inventory Search Tool Wallops Flight Facility Web Service Definition Language Year To Date 26
- Slides: 26