Overview of NASAs Earth Observing Data and Information
Overview of NASA’s Earth Observing Data and Information System (EOSDIS) Managed by the Earth Science Data & Information Systems Project Code 423 May 20, 2016
Earth Science Data Systems Mission Statement The Earth Science Data System Program oversees the lifecycle of earth science data with the principle goal of maximizing the scientific return from NASA’s missions and experiments for research and applied scientists, decision makers and society at large. 2
Extensive Data Collection EOSDIS data collection includes over 6800 data types • Land » Cover & Usage » Surface temperature » Soil moisture » Surface topography • Atmosphere » Winds & Precipitation » Aerosols & Clouds » Temperature & Humidity » Solar radiation • Ocean Dynamics » Surface temperature » Surface wind fields & Heat flux » Surface topography » Ocean color • Cryosphere » Sea/Land Ice & Snow Cover • Human Dimensions » Population & Land Use » Human & Environmental Health » Ecosystems 3
Data Sources Type Satellite/on-orbit Missions Airborne missions In Situ Measurement missions Example Missions Terra, Aqua, Aura, Suomi-NPP, SORCE, GPM, GRACE, Cloud. Sat, CALIPSO, etc. Ice. Bridge, Earth Ventures (5+ missions), UAVSAR, etc. Field campaigns on land (e. g. , LBA-ECO) and in the ocean (e. g. , SPURS) Applications support Near-real time creation and distribution of selected products for applications communities Earth Science Research support Research products from efforts like MEa. SUREs. This also includes data from older, heritage missions (prior to EOS Program) that the DAACs rescue – e. g. , Nimbus, Sea. Sat 4
ESMO ESDIS Mission Operations Flight Operations, Data Capture, Initial Processing, Backup Archive Data Acquisition Science Operations Data Science Data Processing, Transport to Data Management, Data Centers/ Interoperable Data SIPSs Archive, and Distribution and Data Access EOSSpacecraft Tracking and Data Relay Satellite (TDRS) Research Education Internet White Sands Complex (WSC) (Search, EOSDIS Order, Data Centers Distribution) EOS Data Operations System (EDOS) Data Processing Value-Added Providers Interagency Data Centers Earth System Models Direct Broadcast (DB) International Partners EOS Polar Ground Stations EOS Operations Center (EOC) Mission Control Network Services Direct Broadcast/ Direct Readout Stations Instrument Teams and Science Investigator-led Processing Systems (SIPSs) Decision Support Systems
National Aeronautics and Space Administration Science Investigator-led Processing Systems Measurements of Pollution in the Troposphere (MOPITT) Visible Infrared Imaging Radiometer Suite (VIIRS) Atmosphere Ozone Monitoring Instrument (OMI) Microwave Limb Sounder (MLS) Tropospheric Emission Spectrometer (TES) MODIS Adaptive Processing System (MODAPS) Advanced Technology Microwave Sounder (ATMS) & Cross-track Infrared Sounder (Cr. IS) Sounder Ocean Data Processing System (OCDPS) Visible Infrared Imaging Radiometer Suite (VIIRS) Land Visible Infrared Imaging Radiometer Ozone Mapping Suite (VIIRS) Profiler Suite Ocean (OMPS) Ozone Advanced Microwave Scanning Radiometer for EOS 2 (AMSR-E/2) SIPSs perform forward processing of standard products, and reprocess data to incorporate algorithm improvements. www. nasa. gov 6
National Aeronautics and Space Administration Discipline-oriented Data Centers ASF SDC SAR Products, Sea Ice, Polar Processes, Geophysics SEDAC Human Interactions, Land Use, Environmental Sustainability, Geospatial Data LP DAAC Surface Reflectance, Land Cover, Vegetation Indices GES DISC Global Precipitation, Solar Irradiance, Atmospheric Composition and Dynamics, Global Modeling NSIDC DAAC Snow and Ice, Cryosphere, Climate Interactions, Sea Ice PO. DAAC CDDIS Gravity, Sea Surface Temperature, Ocean Winds, Topography, Circulation & Currents Space Geodesy, Solid Earth La. RC ASDC GHRC DAAC Hydrologic Cycle, Severe Weather Interactions, Lightning, Atmospheric Convection Radiation Budget, Clouds, Aerosols, Tropospheric Chemistry OBPG Ocean Biology, Sea Surface Temperature MODAPS/ LAADS MODIS Level-1 and Atmosphere Data Products ORNL DAAC Biogeochemical Dynamics, Ecological Data, Environmental Processes www. nasa. gov 7
Role of EOSDIS DAACs were established based on Earth science disciplines, and provide unique support and expert services to their user communities. • DAACs were originally selected by NASA’s Mission to Planet Earth Program, based on the expertise and heritage of their host organizations. They also provide data and services to the research community for comprehensive, cross-discipline studies needed to understand Earth as an interrelated system. DAAC Functions • • Ingest Level 0 data from EDOS Ingest higher level products produced by SIPS Perform processing of higher level products in some cases Archive assigned data sets Export metadata to CMR Provide user interfaces, tools, and services Distribute data to users (primarily electronically) Provide metrics data to ESDIS Metrics System (EMS) DAACs ensure safe stewardship of NASA’s data 8
DAAC Context Acquisition, e. g. , EOS Data Operations System (EDOS) NASA Sentinel Gateway Science Researchers Processing NOAA NCEP Precipitation Precise Orbit Determination ata e D Educational Users Common Metadata Repository (CMR) cs el ri et ut O t pu Land Data Assimilation Systems od Global Modeling and Assimilation EOSDIS Core System / Discipline specific systems Applications Users M Models, e. g. , y illar Anc ata D Ancillary, e. g. , Me tad ata Science Computing Facilities (SCFs) Distributed Active Archive Center Sci enc Mission Processing Systems Raw Data & ata D cts ce du ien Pro Sc ta Da Science Investigatorled Processing Systems (SIPSs) M EOS Metrics System (EMS) 9
EOSDIS-wide Elements and Capabilities Earth. Data: the EOSDIS website (Earthdata. nasa. gov) provides a focal point for cross-DAAC and EOSDIS content and news sharing, and access to Earth science data and services Common Metadata Repository (CMR): metadata catalog of NASA's EOS data and related data services (e. g. reformatting, pattern recognition) as well as Collection-level metadata for ~ 25, 000 data sets from broad international community. Essential underlying component for data discovery and access. Earth. Data Search Client (EDSC): easy-to-use access to EOSDIS services for Earth science data discovery, filtering, visualization, allowing users to download and access Earth observation data Near Real-Time Capabilities: Provided by “LANCE” (Land Atmosphere Near real-time Capability for EOS) which produces products within < 3 hours of observation. Near real-time capabilities are co-located with the standard science production facilities. Global Image Browse System / World. View: GIBS provides access to full resolution imagery derived from NASA products; Worldview Client allows users to explore GIBS imagery in a Google. Maps like manner. ESDIS Metrics System (EMS): collects and reports on data ingest, production, archive, and distribution across all EOSDIS data centers Earthdata Log-in (User Registration System): provides a centralized and simplified mechanism for user registration and account management for all EOSDIS system components. 10
15+ years of Earth Science Data Total EOSDIS Accumulated Data Archive Volume (Petabytes) Addition of AQUARIUS and ICEBRIDGE products Landsat-7 Data Migrated from LPDAAC to USGS Addition of ALOS PALSAR & UAVSAR products Addition of SNPP CERES&OMPS Deletion of MODIS Collection 4 Products TERRA AQUA ICESat AURA MEASURES 2006 MEASURES 2012 11
EOSDIS Products Delivered: FY 00 thru Apr 2016 129. 2 M Year Pre-decisional - For NASA 12
FY 2015 EOSDIS Number of Data Products Distributed by Country 13
EOSDIS Mission Support CY 2015 2016 2017 2018 2019 2020 2021 Sentinel 1, 3, 5 OCO-3/GESDISC 1/31/17 7/31/17 ECOSTRESS/LPDA AC 8/31/17 SAGE-III /La. RC 7/31/16 1/31/17 CYGNSS/JPL-PO 10/31/16 ICESat-2/NSIDC 4/30/17 10/31/17 4/30/18 GRACE FO/PO ISS-LIS/GHRC 2/29/18 8/31/18 11/21/16 5/31/17 TSIS-1/GESDISC 6/30/18 12/31/18 TEMPO/La. RC 12/30/21 SWOT/PODAAC 10/31/20 4/30/21 NISAR/ASF EVS-2: OMG, NAAMES, ORACLES, Atom, Act-America 12/31/20 6/30/21 Data delivery expected at End of EVS-2 Missions Launch Readiness Date Launch Planned Data Released 14
Earth Science Data & Information System (ESDIS) Project Code 423 Project Manager Dawn Lowe Deputy Project Manager Andy Mitchell Deputy Project Manager/Operations Jeanne Behnke Deputy Project Manager/Technical Mark Mc. Inerney Deputy Project Manager/Resources Diane Hronek ESDIS ISSO Clayton Sigman 585 System Manager Karen Michael 586 System Architect Chris Lynnes Financial Manager Chikia Barnes Katie Baynes 586 Resource Analysts Felipe Romo Ben Hall SNPP/Science Data Systems (SDS) Mgr Alfreda Hall 586 Resource Support Spec. Kathi Thomas Science Systems Dev. Manager/EED 2 COR Tracy Zeiler 586 Science Operations Manager Drew Kittel 586 Deputy SSD Manager Steve Berrick 586 System Engineering Development & Sustaining Eng. Data Visualization Lead Ryan Boller Evolution Dev. Lead Evelyn Ho HW Systems Engineer vacant ECHO Dev & Ops Luther Lighty Deputy SOO Mgr. vacant Kevin Kranacs Nate James 585 Frank Lindsay vacant 586 DAAC Operations Rachel Adolphus SW Development Eunice Eng 586 Network System Engineer May 2016
Earth Science & Data Systems Applications Socioeconomic Data & Applications Center • C. Dobson 09/16/2015 Astrophysics NASA Headquarters Science Mission Directorate (SMD) J. Grunsfeld Heliophysics Earth Science M. Freilich Planetary SAR Systems • C. Dobson Applied Science Application Scientists L. Friedl Program Scientist for EOSDIS L. Tsaoussi Research J. Kaye Land Processes • W. Turner Program Scientists Program Executive for Operating Missions C. Yuhas Flight Programs Atmospheric Dynamics K. Murphy • R. Kakar . . . 400 Flight Projects 420 Earth Science J. Gramling D. Mitchell . . . • K. Jucks Data and Information System (ESDIS) Project D. Lowe SED • D. Green LANCE Physical Oceanography • E. Lindstrom Physical Oceanography DAAC Terrestrial Ecology • H. Margolis ORNL DAAC • B. Lefer Data Center Operations Atmospheric Sciences Data Center L 1 and Atmos. Data • Hal Maring . . . Near Real-time Applications GSFC Earth Sciences DISC Atmospheric Radiation Science Operations Management J. Behnke D. Kittel 600 Global Hydrology Resource Center Upper Atmosphere Research Earth Science 423 500 AETD Crustal Dynamics DIS • B. Phillips Program Executive for Earth Science Data Systems Goddard Space Flight Center (GSFC) C. Scolese Land Processes DAAC Earth Surface and Interior E. Ianson . . . ASF SAR Data Center Ocean Biology & Biogeochemistry • P. Bontempi Ocean Biology DAAC Cryosphere Science • T. Wagner National Snow and Ice Data Center L 1 Atmosphere Archive & Distribution System
ESDIS Budget Execution Across NASA Centers Budget Elements Project Mgmt Systems Engineering DAACs Networks SIPS Sustaining Engineering SNPP BEDI Sentinel GSFC GSFC GSFC MSFC La. RC MSFC ARC La. RC JPL MSFC Lead & Performing Center GSFC JPL ESDIS Code 423 • • • GSFC CS Labor Code 423 CS Travel Code 423 IDIQ Contracts Code 423 Cost Reimbursable Contracts Code 423 Interagency Agreements Code 423 GWAC Code 423 Cooperative Agreement Tasks on Center-wide Contracts Work on Shared Tasks Work on Agency-wide Contracts Misc Procurements • • • Code 600 Tasks on Center-wide Contracts Grants Code 600 CS Travel Code 600 Operating Account Work on Shared Tasks Misc Procurements La. RC • • • Tasks on Center-wide Contract Task on JSC Contract CS Labor CS Travel Misc Procurements MSFC • • • Center-wide Cooperative Agreement CS Labor CS Travel JPL • Tasks on Agency Contract
Global Imagery Browse Services and Worldview https: //earthdata. nasa. gov/gibs https: //earthdata. nasa. gov/worldview Driving Goal: To transform how users interact with and discover NASA Earth data: make it visual Approach: Open-Access Servers – The Global Imagery Browse Services (GIBS) provide open access to full resolution imagery derived from NASA products as web services to mapping clients and scripts – Worldview is an open source, browser -based client to interactively explore GIBS imagery and download the underlying data – 745 K unique users – Distributed 18 TB of imagery to date Client 18
Acronym List A&A AAAS AAG ACRIM ACSI ADEOS AETD AIRS ALOS AMASS AMSR ANGe API ARRA ASCI ASDC ASF ASTER GDS ATMS ATO BGP BMGT Bo. E CARS CDDIS CEOS CERES CF CLASS CM CMR CNES CO COFUR COMET COTS Cr. LS CSOC CSW CWIC CYGNSS C 1 DAAC DAP DAS DEM DISC DOE Authorization and Accreditation American Association for Advancement of Science Association of American Geographers Active Cavity Radiometer Irradiance Monitor American Customer Satisfaction Index Advanced Earth Observing System (Japan) Applied Engineering and Technology Directorate Atmospheric Infrared Sounder Advanced Land Observation Satellite Archival Management and Storage System American Meteorological Society Advanced Microwave Scanning Radiometer Archive Next Generation Application Programming Interface American Recovery and Reinvestment Act American Survey Customer Index Atmospheric Sciences Data Center (Langley) Alaska SAR Facility Advanced Spaceborne Thermal Emission and Reflection Radiometer Ground Data System Advanced Technology Microwave Sounder Authorization to Operate Border Gateway Protocol Bulk Metadata Generation Tool Basis of Estimate Climate Analysis Research Systems Crustal Dynamics Data Information System Committee on Earth Observations Satellites Cloud’s and Earth’s Radiant Energy System Climate and Forecasts Communication Link Analysis Simulation System Comprehensive Large Array-data Stewardship System Configuration Management Common Metadata Repository Centre National D’etudes Spatiales (France) Carbon Monoxide Cost of Filling User Request COnfiguration Management EOS Tool Commercial Off the Shelf Cross-Track Infrared Sounder Consolidated Space Operations Center Catalog Services for the Web CEOS WGISS Integrated Catalog Cyclone Global Navigation Satellite System Configuration 1 Distributed Active Archive Center Delivered Algorithm Package Data Assimilation System Digital Elevation Model Data and Information Services Center Department of Energy DR DRN DSCOVR Ebnet ECC ECHO ECS EDC EDG EDOS EDR EED EDSC EGS ELV EMD EMOS EMS EOC EOSDIS EPGN EPGS ERG ESDIS ESDSWG ESDT ESE ESIP ESISS ESRI ESSAAC ETE ETS EWOC FDS FMU FOS FOR FTP G & A GCMD GDAAC GDS GDEM GEOS GES GHRC GIBS Discrepancy Report Data Ready Notification Deep Space Climate Observatory EOS Backbone Network Earthdata Code Collaborative EOSDIS Clearing House EOSDIS Core System EROS Data Center EOS Data Gateway EOS Data & Operations System Environmental Data Records EOSDIS Evolution and Development Earth Data Search Client EOS Ground System Expendable Launch Vehicle EOSDIS Core System Maintenance and Development EOS Mission Operation System EOSDIS Metrics System EOS Operations Center Earth Observing System EOS Data & Information System EOS Polar Ground Network EOS Polar Ground Station EOSDIS Review Group Earth Science Data & Information System Earth Science Data Systems Working Group Earth Science Data Types Earth Science Enterprise Earth Science Information Partner ESA Space Information Systems Environmental Systems Research Institute Earth System Science and Applications Advisory Committee End to End EOS Test System ECHO Web Service Definition Language Ordering Component Flight Dynamics System Formatter/Multiplexor Unit Flight Operations Segment Flight Operation Review File Transfer Protocol General and Administrative Global Change Master Directory Goddard DAAC Ground Data System Global Digital Elevation Map Goddard Earth Observing System Goddard Earth Sciences Global Hydrology Research Center Global Imagery Browse Services 19
Acronym List GIM GLAS GMAO GN GOBBS GOE GPM GPP GPRA GRACE FO GRAVITE Geomatics Info. Magazine Geoscience Laser Altimeter System Global Modeling and Assimilation Office Ground Network Goddard Opportunities Bulletin Board System Ground Operations Exercise Global Precipitation Measurement Gross Primary Production Government Performance Results Act Gravity Recovery and Climate Experiment-Follow-On Government Resource for Algorithm Verification, Independent Testing, and Evaluation GSFC Goddard Space Flight Center GSIF Ground Station Interface Facility HDF Hierarchical Data Format HEG Hierarchical Data Format - Earth Observing System-Geographic Tagged Image File Format HIRDLS High-Resolution Dynamics Limb Sounder ICD Interface Control Document ICE Integrated Communication Effort ICESat Ice Cloud and Land Elevation Satellite IDN International Directory Network IDPS Interface Data Processing Segment IPR Initial Project Review IPY International Polar Year IRD Interface Requirements Document ISD Information System Division ISO Internal Organization for Standardization ISS-Rapid. Scat International Space Station Rapid Scat-terometer IST Instrument Support Terminal IT Instrument Team IT Information Technology IV&V Independent Verification & Validation I&T Integration & Test JPIP Joint Project Implementation Plan JPL Jet Propulsion Laboratory JPSS Joint Polar Satellite System JSC Johnson Space Center KNMI Koninklijk Nederlands Meteorologisch Instituut L-7 Landsat-7 - Land Remote Sensing Satellite LAADS Level 1 and Atmosphere Archive and Distribution System LAI Leaf Area Index LANCE Land Atmosphere Near real time Capability for EOS La. RC Langley Research Center La. TIS La. RC DAAC TRMM Information System LBA Large-Scale Biosphere Atmosphere Experiment in Amazonia LDB Long Duration Balloon LEOP Launch and Early Orbit Phase LP LPDC LRD LTO MD 5 MEa. SUREs METI MISR MLS MODAPS MODIS MOPFAS MOPITT MOR MOSS MPS MRR MSS NASDA NCAR NCCS NCDC NCR NCSA NEAT NST 4 NICSE NISN NIVR NOAA NPP NRC NRTPS NSOF NSIDC OB. DAAC OBPG OCDPS OCO ODPS OECD OES OGC OHCM Land Processes (DAAC) Land Processes Data Center Launch Readiness Date Linear Tape-Open Message-Digest algorithm Making Earth Science Records for Use in Research Environments Ministry of Economy, Trade and Industry Multi-Angle Imaging Spectro-Radiometer Microwave Limb Sounder MODIS Adaptive Processing System Moderate Resolution Imaging Spectro-Radiometer MOPITT operational fast forward model Measurements of Pollution in the Troposphere Mission Operations Review Mission Operations and Science System Mission Planning System Mission Readiness Review Management Subsystem National Space Development Agency (Japan) National Center for Atmospheric Research NASA Center for Climate Simulation National Climatic Data Center Nonconformance Report National Center for Supercomputer Applications Network Evolution and Architecture Transformation NPP Compatibility Test 4 NPP Instrument Calibration and Science Element NASA Integrated Support Network Netherlands Agency for Aerospace Programs New Obligation Authority National Oceanic and Atmospheric Administration Net Primary Production NPOESS Preparatory Project National Research Council Near Real-Time Processing System National Satellite Operations Facility NASA Science Internet National Snow and Ice Data Center Ocean Biology Distributed Active Archive Center Ocean Biology Processing Group Ocean Color Data Processing System Orbiting Carbon Observatory OMI Dutch Processing System Organization for Economic Co-Operation and Development Office of Earth Science (Code Y ) Open Geospatial Consortium Office of Human Capital Management 20
Acronym List OMAERO OMBRO OMCLDRR OMHCHO OMIDAPS OMOCLO OMPS OMTO 3 ORNL ORR PAD PALSAR PEATE PDR PDS PGEs PODAAC POP PPBE RAID RCN RDS REASon RF RFA RFP RIP ROSES RRR S 4 PA S 4 PM SAGE III SAN SAR SD 3 E SDPS SDS SEB SEDAC SI&T SIPS SLOC SMAP OMI standard product (Aerosol product using the multi-wavelength algorithm) OMI standard product (Bromine Oxide) OMI standard product (Cloud Pressure and Fraction-Raman Scattering) OMI standard product (Formaldehyde) Ozone Monitoring Instrument Data Processing System OMI standard product (Chlorine Oxide) Ozone Mapping and Profiler Suite OMI standard product (Total Column Ozone) Oak Ridge National Laboratory (DAAC) Operations Readiness Review Precision Attitude Determination Phased Array L-band SAR Product Evaluation and Analysis Tool Element Preliminary Design Review Production Data Set Product Generation Executives Physical Oceanography Distributed Active Archive Center Period of Performance Planning, Programming, Budgeting, and Execution Redundant Array of Inexpensive Disks Receipt Confirmation Notification Remote Data Storage Research, Education, and Applications Solutions Network Radio Frequency Request for Action Request for Proposal Retained Intermediate Product Research Opportunities in Space and Earth Sciences Release Readiness Review Simple, Scalable, Script-Based, Science Processing Archive Single Scalable Script-based Science Processor-mission Stratospheric Aerosols and Gas Experiment Storage Area Network Synthetic Aperture Radar Science Data Distribution and Depository Science Data Processing Segment Science Data Segment Source Evaluation Board Socio-Economic Data and Applications Center (CIESIN) System Integration and Test Science Investigator-led Processing Systems Single Lines of Code Safety and Mission Assurance Soil Moisture Active Passive SNF SNPP SOMO SORCE SORR SOW SPo. RT SRB SRR SSDO SSI&T SSR SST S/W SIMBIOS TADS TB TC IDEA TDRSS TES TIM TOTS TLM TRMM TRR TSDIS UPN UPS URS USGS VATC VIIRS WDS WGISS WIST WFF WSDL YTD Superior National Forest Suomi National Polar-orbiting Partnership Space Operations Management Office Solar Radiation Climate Experiment Segment Operational Readiness Review Statement of Work Short-term Prediction & Research Transition system at MSFC Storage Resource Broker System Requirements Review Science System Development Office Science S/W Integration and Test Solid State Recorder Sea Surface Temperature Software Sensor Intercomparison and Merger for Biological and Interdisciplinary Oceanic Studies Terra and Ancillary Data Support Terabyte Tropical Cyclone Integrated Data Exchange and Analysis System Tracking and Data Relay Satellite System Tropospheric Emission Spectrometer Technical Interchange Meeting Transportable Orbital Tracking System Telemetry (subsystem) Tropical Rainfall Measuring Mission Test Readiness Review TRMM Science Data & Information System Unique Project Identifier Uninterruptible Power Supply User Registration System United States Geological Survey Verification and Acceptance Test Center Visible Infrared Imager Radiometer Suite World Data System Working Group on Information Systems and Services Warehouse Inventory Search Tool Wallops Flight Facility Web Service Definition Language Year To Date 21
EOSDIS Cloud Roadmap
EOSDIS Vision 2020 (cloud relevant) Discovery and Access Usage ●Machine Level Discovery and Access for all data ●Intelligent Tool Catalogs suggest tools to work with the data. ●Seamless Cross-agency Discovery. ●Dataset Selection Guidance based on fitness for purpose. ●Publications are linked to data and tools that allow interactions with the data. ●Metadata Naming Conventions for Variables, Platforms, Instruments, Resolution… ●Automatic Mobile Data and Processing to achieve optimal performance. ●Virtual Collections oriented around science problems. ●Quantitative Quality for all data Integration ●Reproducibility: of research results with high precision. ●NASA data can be combined with data from other agencies and nations ●Concise, Comprehensive and Consistent documentation for all data variables. ●Tools and services within the community are easy to combine. ●Capacity Building mechanisms for people with limited literacy in science, technology, and/or English. ●Any scientific resource can be shared ●Data Analysis at Scale over any arbitrarily defined area. ●Dataset Upgrading for High-value datasets to fully support rich capabilities
Cloud Prototypes 3 1 2 Key Archive Mgmt Analytics Support Application Hosting 24
1 Archive and Distribution Prototypes Benefits • • Cost • Modest savings on storage • Smoother multi-year cost profile Science • Easier data merging, fusion, and intercomparison 25
1 Archive and Distribution Prototypes 1. Alaska Satellite Facility DAAC prototype: store and distribute Sentinel radar data with Amazon storage 2. ARIA radar processing prototype • Data storage component • On-demand processing component 3. Ingest and Archive management prototype 26
1 Data Management Paradigm Shift ESDIS BUYS & PROVISIONS STORAGE 1 DAAC PROVIDES ARCHIVING SERVICES FOR A DATA PRODUCT ANY DAAC CAN ADD SERVICES TO ANY PRODUCT TO SERVE THEIR USER COMMUNITY NASA Domain and Data Stewardship Security Policies Community Service Cloud Archive ESDIS Provisioning 27
BUT. . . BUT: HAVE TO SOLVE THE DOWNLOAD CHARGE POLICY FOR THE AVERAGE SCIENTIST! • Egress is pricy using current structure • But • Intra-cloud transfers can be free • Egress to Internet-2 is free NEEDS USER PARADIGM SHIFT AS WELL TO REALIZE FULL POTENTIAL NETWORK TRUST ISSUES LOOM ON THE ROAD TO DEPLOYMENT
2 Cloud Analytics Prototypes Benefit: Large-scale analysis enabled by cloudbased parallelization is the killer app for cloud computing 29
2 Cloud Analytics Prototypes ANALYSIS SUPPORT TOOLBOX TO ATTRACT USERS TO CLOUD ANALYTICS Analytics Optimized Storage for Big Data Analytics 30
2 Analysis in the Cloud LONG TERM PARADIGM SHIFT: HOW SCIENTISTS WORK ON DATA • Scientists will work on data “in place” • instead of downloading • High-value data will (also) be in databases • Pre-existing toolsets will be easy to find and use BUT THE PARADIGM SHIFT NEEDS USER PARTICIPATION U s e r s E S D I S + � D A A 31
“Analysis” More Complexity Subset Data Variable Spatial Area Reprojection Quality Filter Mosaicking Transform Analyze Simple Stats Complex Stats End User’s Algorithm Seasonal Time Series 32
2 Cloud Analytics Prototypes DAAC Cloud Analysis Toolbox to Enable Earth Science DAAC ESDIS Open. Source 33
2 Cloud Analytics Prototypes Analytics Optimized Storage for Big Data Analytics 34
Cloud Analytics-Optimized Storage Prototype NEXUS: DATA ANALYSIS SERVICE BASED ON CLOUD-COMPATIBLE TECHNOLOGIES • Map. Reduce algorithms • Cassandra database for data PLAN 1. BENCHMARK: COMPARE PARALLELIZED NEXUS ANALYSIS WITH CURRENT SINGLE-THREADED ANALYSIS 2. CLOUDIFY: MIGRATE NEXUS TO AMAZON WEB SERVICES a. Add to CATEES toolkit 3. MIGRATE NEXUS ON TOP OF NASA-COMPLIANT GENERAL APPLICATION PLATFORM
3 Application-Hosting Prototypes Benefits Cost: Smaller custom codebase -> Modest savings on maintenance Hardware buy avoidance, smoother cost profile over time Performance Speed Reliability and Availability 36
3 Application-Hosting Prototypes Cloud Application Migration Port existing applications to public cloud Leverage NASA-compliant General Application Platform (NGAP). . . 37
NGAP: “Compliance-as-a-Service” Software-as-a-Service Compliance-as-a-Service security controls, Authorization to Operate governance procurement and accounting reliability and availability Platform-as-a-Service Infrastructure-as-a-Service 38
3 Application-Hosting in the Cloud PARADIGM SHIFT: HOW WE IMPLEMENT SYSTEMS • Off-the-shelf systems reduce effort for hardware procurement and deployment • Automate everything: testing, deployment, scaling, failover, … • Write only domain-specific code 39
Roadmap Prototypes Year 1 Year 2 Long Term Goal Archive and Distribution • ASF DAAC Web Object Storage • Ingest/Archive prototype Value-added Processing Operational EOSDIS Archive in the Cloud Analytics Proofs of Concept Public (Beta) Prototypes Operational Support of Cloud Analytics Application Hosting and Processing • GIBS All EOSDIS Common • Earthdata Login Services in the Cloud • ECC • Earthdata Search • CMR • GIBS study 40
EOSDIS Cloud Status. . . Prototypes Status Archive and Distribution • ASF DAAC Web Object Storage is well underway • Ingest/Archive prototype has been scoped • Contractors have been tasked with adapting subsetting service to work with AWS S 3 storage Cloud Analytics • DAACs+ESDIS have begun work on cloud analysis toolkit • NEXUS prototype has begun Application Hosting • • NGAP is authorized to operate under OCIO’s ATO Earthdata Search was developed in the cloud CMR migration study is complete GIBS study is complete 41
A Lot of Questions… 42
Q: How can we supply data to all users on a non-discriminatory basis? Background: egress charges depend on user demand could be unbounded. Egress to the same cloud zone is free for users using the (same) cloud for analysis, as is egress to Internet-2. But this still A: Egress charge structures are under negotiation between HQ and AWS A: In the end, a data routing server could be used… 43
Q: How can we avoid vendor lock-in? • Storage • On-premise backup? • Or: Cold storage backup in alternate vendor? • Innovative services (e. g. , AWS Lambda) • Trade studies on cost v. risk 44
Q: How can we attract end users to the cloud, and how will that alter usage patterns? A: Cloud analytics support toolbox + outreach 45
Q: How can we predict pricing 2 -5 years out… . . . especially as usage patterns change A: Prototypes, studies, still working on this. . . 46
Big Questions 5. HOW CAN WE MIGRATE NEAR-ARCHIVE DATA SERVICES TO WEB OBJECT STORAGE? OPe. NDAP w 10 n Web Object Storage OGC W*S 47
Q: What functionality or data should NOT go into the cloud? A: TBD. 48
Q: Do we need new operations policies or procedures? A: Most Likely. The prototypes will tell… 49
Q: Who Owns the Data? A: NASA AWS Response: As between AWS and its customer, the customer owns all right, title, and interest in and to all content that the customer or its end users (i) run on the AWS services, (ii) cause to interface with the AWS services, or (iii) upload to the AWS services under the customer’s account or otherwise transfer, process, use, or store in connection with the customer’s account (we call this “Customer Content”). AWS does not use or disclose Customer Content for any purpose other than (i) as necessary to provide the AWS Service offerings to customer and its end users or (ii) as is necessary for AWS to comply with the law (e. g. , to comply with any request of a governmental or regulatory body including, for instance, subpoenas or court orders). AWS’s terms do not grant AWS the right to copy and sell Customer Content, whether in isolation or as part of a “value added” solution. Note, however, that NASA may want to verify whether its agreement(s) with its chosen systems integrator or AWS value added reseller grant such a right vis a vis NASA and the systems integrator or AWS value added reseller. AWS would not be privy to those upstream agreements. Importantly, the customer manages and maintains ownership over Customer Content. With AWS, customers: Determine where their customer content will be stored, including the type of storage and geographic region of that storage. The customer selects the geographic location where Customer Content is stored and AWS will not move Customer Content except as the customer authorizes or as necessary to comply with the law. Choose the secured state of their customer content. AWS offers customers strong encryption for customer content in transit or at rest, and AWS provides customers with the option to manage their own encryption keys. Manage access to their customer content and AWS services and resources through users, groups, permissions and credentials that the customer controls. So, for instance, NASA could set up permissions as to customer content that it would want its end users to have access to. More information at: https: //aws. amazon. com/compliance/data-privacy-faq/
Q: Who monitors proprietary claims? A: Proprietary claims w. r. t. content are monitored by NASA (but are not expected given the data are already publicly available. ) AWS Response: AWS does not monitor or control the content that a customer or its end users upload to the AWS service. Accordingly, this content is the customer’s responsibility. AWS may disable content that is illegal or violates the AWS Acceptable Use Policy found here: https: //aws. amazon. com/aup/ Regarding proprietary claims, the AWS customer monitors proprietary claims concerning any Customer Content. In NASA’s case, this could be NASA’s CIO Office or, perhaps, a systems integrator if NASA opts to use one to manage its use of AWS services. Monitoring Customer Content would include, but would not be limited to, properly handling and processing notices sent to the customer by any person claiming that the customer content violates such person’s rights (including notices pursuant to the Digital Millennium Copyright Act) and ensuring compliance with AWS’s Acceptable Use Policy here: https: //aws. amazon. com/aup/.
Q: What analytics of users, marketing, etc. would be permitted? A: None. AWS Response See AWS’s response to the question on data ownership. AWS does not use Customer Content nor does it derive information from it for AWS’s own marketing or advertising purposes.
Q: What advertising/marketing would be permitted? A: To be negotiated with NASA HQ AWS Response: AWS does not use Customer Content, or information derived from such content (such as, for instance, End User data) for marketing or advertising purposes. AWS’s standard terms do allow AWS to use the Customer’s name as a reference, but for U. S. Government customers we understand that this is problematic. Our practice is to negotiate with, the U. S. Government customer before using the customer’s name for advertising/marketing purposes.
Q: Do we have to recompete the contracts every 5 years or so? A: Maybe, but this issue is really one for the NASA OCIO
Q: Are there terms and conditions about having scrubbed all the data when we leave a company? A: Yes: customer deletes content, and storage devices are reclaimed. When decommissioned, they are sanitized a la NIST-800 -88. N. B. : All EOSDIS data in the cloud will be Low Sensitivity data. AWS Response Security and compliance responsibilities are shared between AWS and the AWS customer. As per the AWS shared responsibility model, the customer retains control and ownership of Customer Content. The customer is responsible for managing encryption, controlling access to, deleting Customer Content. This includes deleting Customer Content when the customer no longer wishes to use the AWS Services. Once a customer deletes Customer Content from the AWS Services, the deleted Customer Content is disabled and the underlying storage device(s) are reclaimed for use by AWS’s systems in accordance with AWS’s standard policies. When a storage device has reached the end of its useful life, AWS procedures include a decommissioning process that is designed to prevent Customer Content from being exposed to unauthorized individuals. AWS uses the guidance detailed in NIST 800 -88 (“Guidelines for Media Sanitization”) as part of the decommissioning process. This includes sanitizing and destroying all decommissioned HDD and SDD storage devices in accordance with industry-standard practices.
Q: Are they free to continue their own version of the service as a for profit activity? While they are under contract to us, are they free to add any algorithms they develop to for-profit services on the side? A: Yes, EOSDIS data are freely available to anyone that wants them, to do whatever they want. But there is no impact to our program from value-adding activities. AWS Response As stated above, AWS customers maintain ownership of their customer content and select which AWS services process, store and host their customer content. AWS does not access or use customer content for any purpose other than as legally required and for maintaining the AWS services and providing them to our customers and their end users. We never use customer content or derive information from it for marketing or advertising. As between you and us, you or your licensors own all right, title, and interest in and to Your Content. Except as provided in this Section 8, we obtain no rights under this Agreement from you or your licensors to Your Content, including any related intellectual property rights. You consent to our use of Your Content to provide the Service Offerings to you and any End Users. We may disclose Your Content to provide the Service Offerings to you or any End Users or to comply with any request of a governmental or regulatory body (including subpoenas or court orders).
EOSDIS Roadmap BACKUP SLIDES 57
Next Generation Application Platform 58
Cloud Roadmap Category Year 1 Activity Year 2 Activity Longer Term Goal Archive and Distribution Prototype Ingest, Index, Archive & Distribution Demonstrated Added Value Processing Broader DAAC and (e. g. RTC) Mission Engagement Data Access Studies and Prototype (e. g. , OPe. NDAP) OPe. NDAP Analysis and Benchmarking Data Egress Routing Object Storage Optimizations and Deployment EOSDIS Services Migration to Cloud (CMR, GIBS, EDSC, Earthdata) EDSC operating in the cloud? CMR in the cloud? GIBS cloud study and pilot(? ) ECC cloud study Operationalize Cloud Prototypes Earthdata Login Routine product deployments of all EOSDIS common services to the cloud via NGAP DAAC tools deployed to cloud via NGAP Analysis Support Proof of Concept EOSDIS-wide Community Curation: Toolbox for End contributions to toolbox more tools, usage recipes, 59 Simplified version Users sample scripts
Big Volume and Distribution Growth Archive Volume Distribution Volume
Analytics-Optimized Storage (Cloud Database) Options SAAS • Google Earth Engine BIGTABLE DATABASES • HBase • Cassandra DATA WAREHOUSES • AWS Redshift • Hive ARRAY DATABASES • Sci. DB • Rasdaman HYBRID APPROACH • net. CDF over Hadoop Distributed File System 61
CLOUD Archive and Distribution Prototypes 1 ASSUMPTIONS • Office of NASA Chief Information Office (OCIO-HQ) commercial cloud available at nominal cost • OCIO assistance in dealing with IT security red tape • DAAC resources available • Budget availability EXPECTATIONS • • Metadata and imagery exports via cloud infrastructure Distribution & usage metrics exports established from cloud At least one subsetting/aggregation service established Lessons-learned documentation 62
2 Cloud Analytics Prototypes ASSUMPTIONS • OCIO commercial cloud available at nominal cost • Leverage current prototype efforts at George Mason University, GSFC and JPL EXPECTATIONS • Demonstrated ability to process datasets at scale • Establish guidelines for optimizing storage of datasets • Which storage scheme • Cloud-vendor-neutral provisioning of analysis tools via containers • End-user uptake of processing in the cloud enabled by easy access to working tools, examples, and recipes 63
3 Application-Hosting & Processing Prototypes ASSUMPTIONS • OCIO commercial cloud available at nominal cost • OCIO assistance in dealing with IT security red tape • Work done under ESDIS local contract EXPECTATIONS • Application Deployment • NGAP* matures as Application Platform • Lessons-learned provided across EOSDIS • Common Services Transition to Cloud NGAP = Next Generation Application Platform 64
- Slides: 64
