GFOI Space Data Services Brian Killough NASA CEOS

GFOI Space Data Services Brian Killough / NASA, CEOS SEO SDCG-8 Session 7, Agenda Items 26 and 27 SDCG-8 DLR, Bonn, Germany September 23 rd-25 th 2015

Agenda • 3 -Year Work Plan Tasks and Outcomes o o o #6: Ensured On-going Coverage (+ Archive Characterization) #7: Interoperable Data Discovery Tools #8: Assembly and Delivery of Core Data #10: Cloud-Computing Pilot Projects #11: Model National GFOI Data Services System • CEOS Data Cube Summary Report - Reported by Matt Jondrow

#6: Ensured On-going Coverage (+ Archive Characterization) • The SEO has completed 22 country reports to characterize Landsat coverage. These were delivered to countries at SDCG and Silva. Carbon meetings. 2 more reports are being delivered at SDCG-8 (Peru and Vietnam). • Feedback from countries has been limited, but those that have responded, were positive. The SEO has provided additional detailed support to countries (Kenya and Colombia) to identify low-cloud scenes and order surface reflectance products. • Future. . . The SEO can develop systems analysis tools, user guides and tutorials to allow countries to do their own analyses. There is also a recent discussion about creating a “GFOI Data Services Portal”. In addition, the SEO can continue to offer custom services.

Country Report Summary • Two detailed “country reports” were completed for SDCG-8 (Peru and Vietnam). • “Country Reports” contain a summary of Landsat acquisitions over the country since 2000 (see example tables/charts on the next page). • The reports include details about cloud cover and data processing levels to allow countries to assess the acceptability of available scenes forest mapping and reporting. • Analyses were completed using the “COVE Coverage Analyzer” tool and a custom queries of the Landsat archive system for data processing status. 4

Example Tables/Charts TOP Table Summary of Landsat 5/7/8 acquisitions per year (2000 through 2014). Middle Table Detailed Path-Row summary of low-cloud (<10%) scenes and medium-cloud (<60%) scenes in 2014 and 2015 for Landsat-7 and Landsat-8. Bottom Chart Landsat acquisitions with <20% cloud cover summarized by year since 2000. SDCG-7 Sydney, Australia 5 March 4 th – 6 th 2015

Coverage Analyzer Example Peru Example • The COVE Coverage Analyzer can easily produce the figure on the left. • These figures are valuable for identifying path-row regions with cloud issues. • Due to extreme cloudiness, there are several regions in the north-west of Peru (Ecaudor border)that did not acquire any images with <20% cloud cover in 2014 summary of total Landsat-7 and Landsat-8 scenes with <20% cloud cover. The maximum for any path-row region is 23*2 = 46 scenes. The south border SDCG-7 Sydney, Australia 6 had 29 scenes in one location. March 4 th – 6 th 2015

#7: Interoperable Data Discovery Tools • The COVE Tool (Coverage Analyzer) now includes Landsat 7/8, SPOT 1 -6, Pleaides -1 A/1 B, Radarsat-2, ALOS-1. • There have been few requests for searches of Envisat, ERS, so adding those archive connections is low priority. • Adding an archive link for Sentinel-1 A and Sentinel-2 A is highly desired, but a mechanism does not exist to get this data from ESA/EC. • The SEO vision is that the COVE tool can provide a “one-stop” location to perform coverage assessments and “discovery” of valid images. Once discovered, users can find links to the data in COVE, or be directed to other sites (CEOS Open. Search from WGISS) to find the data. • Data discovery is also part of the (Space Data Management System (SDMS) scene -based tools.

#8: Assembly/Delivery of Core Data • The new Global Data Flow Study is focused on this topic. (see additional chart). • There are 3 methods of data delivery (business-as-usual data over the internet or on drives, delivery using cloud-computing, or delivery in-country using Data Cubes (see additional chart). • After discussion with FAO at SDCG-7, the SEO took an action to develop a “GFOI Space Data Guide” that provides details and contacts for data access, details and contacts for coverage analyses (SEO), and links to resources (COVE, tutorials).

The scale of the data problem • Kenya: 4 TB of Landsat 7/8 data since 2000. Adding 4. 5 TB per year for Landsat and Sentinel -2 A. Storage needs are ~22 TB for 3 years. Cost of cloud-based storage (Amazon EBS) and processing (Amazon EC 2) is ~$29, 000 U. S. per year. • Colombia: 11 TB of Landsat 7/8 data since 2000. Adding 8 TB per year for Landsat and Sentinel-2 A. Storage needs are ~42 TB for 3 years. Cost of cloud-based storage and processing is ~$53, 000 U. S. per year. • 70 GFOI Countries: Consider all Landsat 7/8 and Sentinel-2 A data = 200 TB per year, 1. 3 PB of new data by 2020. This does not count historic data needed for baseline analyses. • 50% of the countries have a required annual data volume of 0. 6 to 3. 4 TB (Landsat and Sentinel, combined). The mean volume is 2. 8 TB and the median volume is 1. 2 TB. We have a BIG data problem with non-sustainable costs. The global data flow study will explore these problems and potential solutions.

Global Data Flow Scenarios Space Agency In-Country Users Satellite Level-1 Data BAU Business-as-Usual Space Agency Processing into Level-2 SR products Scenario-2 Satellite Analysis Ready Data (ARD) Products Scenario-1 Intermediate Data Storage and Processing (Cloud-based or Data Hubs) • Scene-based tools • Hosted Data Cube and tools • Hard Drive Delivery or Internet Download • Countries perform processing and data analysis locally • Hard drive delivery or internet download • Countries perform data analysis on ARD locally • Countries use scene-based tools or a Data Cube (countries create their own Data Cubes using CEOS open source tools or utilize CEOS support to initialize a baseline Data Cube) • Country analyses of ARD performed on Cloud or Data Hub • Countries download analysis products via Internet

#10: Cloud Computing Pilot Projects • • KSAT/AMA delivered a scene-based SDMS to FAO on November 5, 2014. FAO has developed a similar tool called “SEPAL” and is using it in-country forest management. The SEO has advanced its scene-based SDMS and will make it available for testing in Kenya and other GFOI countries. The SEO delivered an SDMS to Colombia (Tan. DEM-X DEM) on December 3, 2014. This data is being used with SRTM forest cover research. The SEO has developed a full-country Kenya Data Cube with Landsat 7/8 data since 2000. This development includes the use of analysis-ready surface reflectance products from USGS, modified ingestion software from GA/CSIRO, and a new reference user interface from AMA. All of the software and tools will be available on an open source site. The SEO is starting a Colombia Data Cube project. By late 2015, a mini-cube will be completed for 4 path-row regions. By the end of 2016, a full-country cube will be completed with multiple data layers and a user interface that supports multiple applications (GFOI and GEOGLAM).

SEO SDMS (scene-based) 12

High-Level Data Cube Requirements • Free and open access to software and APIs • Documented processes for data cube generation, new data ingestion, and application user interfaces • Cloud-based or local deployment • Use of “Analysis Ready” data products • Preservation of native data product grid formats • Computational flexibility to allow reprojection into “nested grid” formats for multiple dataset interoperability and spatial consistency • Baseline user interface that supports data cube statistics/analysis and optical image preparation (e. g. mosaics). • Enable user development of applications through flexible APIs • Architecture flexibility and standards to support Data Cubes for local, regional or country scales

Data Cube Datasets (priority order of addition) • Landsat-5/7/8 (Surface Reflectance) from USGS. Products available, by order. • SRTM. . . NASA Space Shuttle Digital Elevation dataset (Feb 2000) now available globally at 1 -arcsec (30 meters) resolution. • ALOS (PALSAR L-Band SAR). Free and open annual mosaics from 2007 -2010 provided by JAXA. Working with Ake Rosenqvist (GFOI) to demo. • MODIS (Terra, Surface Reflectance, 8 -day Level-3 500 -m Global, MOD 09 A 1). Working with Alyssa Whitcraft (GEOGLAM) to demo. Also will work with GA/CSIRO to utilize their experience with this dataset. • SPOT-5 (10 -meter multi-spectral). Plans to obtain SPOT-5 Take-5 data over Kenya sample site (processing complete in Sept 2015). Over time, CNES plans to process/release all data. • Sentinel-1 A: C-band SAR data, plans to add sample images to prototype in late 2015 • Sentinel-2 A: Launched June 21, data available in late 2015 • Rapid. Eye: SDCG-8 action to obtain sample data for Data Cube ingestion testing. • In-Situ Data: Working with Kenya SLEEK team to investigate approaches for adding in-situ climate data. May also consider global rain gauges and other ground-based data for demo. • Radarsat-2 (C-Band SAR): Dataset is restricted, so we are not planning to add this data to any of the prototypes. We will consider adding accommodations to the ingester to support this dataset, so others can use the data.

#11: Model National GFOI Data Services • It is expected that the prototypes in Kenya and Colombia would be ideal testing locations toward a model national GFOI data services system. • FAO will be using SEPAL in many countries, so feedback from its use would help support requirements for a model national system. QUESTIONS • What is the process for integration with the MGD? • Is it possible to have a common country engagement strategy among GFOI and FAO? For example, where do we use SEPAL, SDMS or Data Cubes? • How can we tailor our data services for the future to better meet the needs of GFOI?