Australian Geoscience Data Cube A Collaboration between Geoscience

Australian Geoscience Data Cube A Collaboration between Geoscience Australia, CSIRO and NCI GA/CSIRO Agency Report Robert Woodcock - CSIRO Simon Oliver – Geoscience Australia CEOS WGISS 41

Overview • Regional Copernicus Data Access/Analysis Hub • RADAR cube • Australian Geoscience Data Cube – Update on progress: • Version 1 User Documentation - development discontinued - focus on v 2 • Version 2 Development – The Future: Data Cubes and WGISS CEOS WGISS 41

Australia’s Regional Copernicus Data Access/Analysis Hub An integrated ‘Team Australia’ approach to All Sentinel Products For Support government information requirements. Support the broader objective of enhancing access to satellite Earth observation data by research, industry and civil society. Facilitate collaboration between Australians, Europeans and inhabitants of the South-East Asia-South Pacific region in exploitation of Earth observation data. Benefits for: Australia, the region, EC/ESA/EUMETSAT, and the global satellite EO community. Supporting Consortium Partners Collaborators CEOS WGISS 41

Australia’s Regional Copernicus Data Access/Analysis Hub ESA Int. Hub EUMETCAST Terrestrial GEANT/AARNET Repository (NCI) Original Format Data Files (HSM Storage) Analysis-Ready Data (Lustre/spinning disk) Interoperable Services (e. g. Thredds) Agreed Standards Regional Portal (a la ‘Sci. Hub’) Master Cloud and HPC (Petascale) Operational Products Deep National Archives Partnerships (pay-per-use) Delivery for: Australia Region R&D for: Research/Academia Civil Society Government Industry Australia and Region Industry Exploitation (Government Systems) CEOS WGISS 41

RADAR cube UK Catapult collaborative agreement signed between: GA / UNSW / CSIRO / UK Catapult Analysis Ready Data for RADAR: • Sentinel-1 o. Level-1 Interferometric Wide Swath o. Level-1 Extra Wide Swath ● ERS-1/2 SAR ● ENVISAT ASAR ● ALOS-1 PALSAR CEOS WGISS 41

AGDC Overview A series of data structures and tools to enable efficient analysis of large earth observation archives in HPC / Cloud and Desktop environments • Simple Data Structures – Spatially regular tiles – Managed by a relational database • Calibrated and Standardised Unique Observations – Surface Reflectance Observations • Quality Assured Observations – Flagged for cloud, cloud shadow, saturation and other quality indicators • Analysis Ready Data as input CEOS WGISS 41

AGDC Programme • • Partners: GA, CSIRO and the NCI International collaborators are increasingly involved including the USGS, NASA and CEOS. – In the midst of moving to an updated version reflecting recent advances in technology – Supporting the establishment of other international data cubes, initially with Kenya and Colombia AGDC is currently supporting a range of remote sensing applications across the water, vegetation and mineral domains Providing valuable information for environmental monitoring and modelling across all Australian jurisdictions CEOS WGISS 41

Landsat Analysis Ready Data preparation Data. Cube CEOS WGISS 41

Using tidal models to map tidal extents Tidal Range of >10 m Tidal Zone Extent Can be attributed with offsets of LAT to lowest observed tide and HAT to highest observed Tidal Zone Morphology Fraction of water observations over the time series. Can we attribute this with depths? CEOS WGISS 41

Using tidal models to map tidal extents Time-series analysis - calculating median from acquisitions at low (right) and high (right) tide models Dr Stephen Sagar CEOS WGISS 41

Mangrove change over 25 years CEOS WGISS 41

Terrestrial applications Vicarious calibration sites • Identify climatic zones, spatial and temporal variation, and seasonal suitability for calibration activities Landsat MODIS blending • Blend Landsat and MODIS scenes to produce Landsat-like data (25 m resolution) with MODIS repeat cycle (~ every 4 -days) GEOGLAM Rangeland Pasture Productivity • Rangeland pasture cover information for ecosystem health and livestock resources globally CEOS WGISS 41

Solutions to assist algal management in lakes and rivers An increasing global problem Systems under pressure Human and animal health impacts AU$180 -240 M per annum in bloom impacts Tim Malthus, Janet Anstee, Hannelie Botha, Stephen Gensemer, Eric Lehmann, Xavier Ho CEOS WGISS 41

Processing Pathway and Decision Support Tools • From remote sensing data to visualization of algal alerts • Select extents from WOFS data • Source latest AGDC dataset(s) • Apply pixel quality and optical filters • Apply algal alert algorithm(s) • Produce visualisation products CEOS WGISS 41

AGDC v 2 • • Complete rewrite of AGDC v 1 (v 2 Deployment July 2016) Reprocessing of Landsat - improved geolocation, spectral & terrain correction MODIS refinements to ARD preparation (C 005 and prep for C 006) AVHRR national archive move to AGDC under investigation • Hybrid No. SQL influenced database model • Ingest support for multiple sensors • Multidimensional storage - Net. CDF 4 -CF 1. 6 • Multidimensional access - XArray and Dask • Analysis and Production Pipeline Provenance capture • Projection support / Geographic / Sinusoidal / Albers Equal Area etc. • Transition planning - Landsat Archive Reprocessing - featuring terrain corrected Landsat as an additional layer • Analytics / Query Language / Quality and Spectral Semantics Test it out - github. com/data-cube/agdc-v 2/tree/develop - example ingest configurations for MODIS MCD 43 A 1 ->4, USGS Landsat SR, Himawari 8 BRF from Australian Bo. M, GA ARG 25 NBAR/T products, SRTM DEM. CEOS WGISS 41

AGDC v 2 Data Ingestion Prepare data for analysis in a couple of steps: 1. Storage configuration - Projection / resampling / grid specification 2. Data preparation (assuming Analysis Ready Data input) - write configuration file allowing datacube to interpret input data 3. Ingest 4. Stack 5. Analyse TODO - simplify data preparation - currently using python template CEOS WGISS 41

AGDC v 2 Projection and Sensor Support Flexible storage unit representation Ability to access data in native or lat/lon projection coordinates through API CEOS WGISS 41

Open Data and Code AGDC Web: http: //www. datacube. org. au AGDC Wiki: http: //www. datacube. org. au/wiki/Main_Page Code repositories are available through Git. Hub: https: //github. com/data-cube/agdc-v 2/tree/develop CEOS WGISS 41

OGC Discrete Global Grid Systems (DGGS) SWG • DGGS an important part of the future versions of Datacube • OGC DGGS SWG to begin elaboration of Extension Standards to the DGGS Core Standard • Anticipated Extensions include: – Interoperability interface protocols for OGC Web Services (e. g. WCS, WCPS, WCTiles, etc…) to facilitate DGGS-to-DGGS communication and processing – 3 D (and higher dimensional) DGGS Specifications – Best Practice Guide CEOS WGISS 41

Future: WGISS involvement • • AWS - S 3 object storage study The Future: Data Cubes and WGISS • Discrete Global Grid Systems - OGC SWG and implications for AGDC evolution CEOS WGISS 41