HMA FollowOn Task 3 Online Data Access ODA

HMA Follow-On Task 3 Online Data Access (ODA) in the frame of ESA's Heterogeneous Missions Accessibility (HMA) initiative Final Presentation ESRIN 20 – 21. June 2011 HMA-FO ODA, June 2011

Agenda EO-WCS Concepts Targets & Achievements • • Requirements & Use Cases New OGC Standards Reference Implementations Alliance with FOSS/Map. Server Deliverables • ECSS Documents • OGC • Software EO-WCS Software • EOx. Server Architecture • EOx. Server Functionality • Web. Map. Viewer Client • • • Overview Definitions Dataset Series - Past Dataset Series - Now Stitched Mosaics – Now Improvements due to WCS and EO-WCS Future • Developments Summary Task-3 Demonstration • Server – Adding Data • Client HMA-FO ODA, June 2011 Slide 2

Targets & Achievements (1) Collect Requirements for Online Data Access § Include GSC-DA issues on how to handle datasets and collections § Identify requirements and scenarios for transactional tasks (adding & updating of mosaics, while keeping track of the history) Requirement Baseline documents including an extensive set of Use Cases have been delivered Review, analyze and update standards addressing coverages § At Kick-Off, a change to the, at that time, premature WCS 2. 0 specification has been requested and was accepted WCS 2. 0 had been finalized, submitted and is now an OGC Implementation Standard HMA-FO ODA, June 2011 Slide 3

Targets & Achievements (2) Define a "WCS Application Profile for EO Sensors" (EO-WCS) Document Submit to OGC and guide its evolution EO-WCS has been defined and Document is submitted as OGC 10 -140 “WCS Earth Observation Application Profile” to OGC OAB Motion (2011 -05 -06): Approve OGC 10 -140 for release for public comment (after applying some editorial changes). HMA-FO ODA, June 2011 Slide 4

Targets & Achievements (3) Demonstrate the proposed specification by extending an Open. Source Software Map. Server has been chosen as platform and has been extended for WCS 2. 0. This extension is included in Map. Server 6. 0 (released: 2011 -05 -12) The EO-WCS has been implemented, based on a modular approach (Unix philosophy: "Write programs that do one thing and do it well. "). EO-WCS Software publicly released as: (http: //eoxserver. org) § Developments are entirely based on Open. Source software § Released under the GNU GPL 3 Open Source license § Newest release is: 0. 1. 2, dated: 2011 -05 -04 § Sustainably maintained through: • Applying policies and procedures as laid out by OSGeo e. g. forming of a PSC (http: //eoxserver. org/doc/en/rfc 0. html) • Subversion code repository (trunk, releases, etc. ) • Trac installation providing issue tracker, wiki, etc. • Mailman mailing lists HMA-FO ODA, June 2011 Slide 5

Targets & Achievements (4) Demonstrate the specification using an updated WCS client Web. Map. Viewer Client has been updated to handle WCS 2. 0 and EO-WCS Set up a demonstration and test server The Demonstration Server (WCS 2. 0 / EO-WCS) at ESA (within a virtual machine) is available at: § http: //ssews 1. esrin. esa. int/wcseo/ows? SERVICE=WCS &VERSION=2. 0. 0 § http: //ssews 1. esrin. esa. int/wcseo-test/ows? SERVICE=WCS &VERSION=2. 0. 0 Perform ATS, ETS and TEAM engine testing WCS 2. 0 Test Suite is available, based on the OGC 09 -147 r 1 Abstract Test Suite (https: //svn. opengeospatial. org/ogc-projects/cite/scripts/wcs/2. 0. 0/) HMA-FO ODA, June 2011 Slide 6

Targets & Achievements (5) Alliance with FOSS/Map. Server Results from participation at Map. Server Code- Sprint • Map. Server community valued the chosen approach of EOx. Server for providing the EO functionality by extending Map. Server (separating imaging engine and metadata handling) • Full integration of EO-WCS functionality into Map. Server was discussed, but is not seen as adequate approach (EO metadata handling too specific) • Map. Server community proposed to host/distribute EOx. Server on Map. Server platform to form a strong alliance. Further discussion at FOSS 4 G (Sept. , Denver) HMA-FO ODA, June 2011 Slide 7

Deliverables - ECSS Documents – HMA Wiki Date Status DJF-Software Implementation – Rationale v. 1. 0 (DJF-TN) 2001 -02 -24 Final DDF-Software User Manual (Client) v. 1. 0 (DDF-SUM) 2011 -01 -25 Final DDF-Software User Manual (Server) v. 1. 0 (DDF-SUM) 2011 -01 -21 Final DJF-Software Verification Report v. 1. 1 (DJF-SVR) 2011 -01 -21 Final DJF-Software Validation Specification v. 1. 2 (DJF-SVS) 2011 -01 -21 Final DDF - Software Design Document v. 1. 0 (DDF-SDD) 2010 -11 -27 Final DJF - Software Reuse File v. 1. 0 (DJF-SRF) 2010 -11 -27 Final TS - Software Requirements Specification v. 1. 2 (TS-SRS) 2010 -11 -27 Final RB - Software System Specification v 1. 4 (RB-SSS) 2010 -11 -04 Final Use Cases applied to WCS 2. 0 EO AP TN 2010 -10 -27 Final RB - Technical Note v 1. 1 2010 -07 -06 Final HMA_Specification_Dependencies_(Cros_linking) 2010 -04 -28 Final Metadata_requirements_1. 0_(RB) Spot Image 2010 -03 -01 Final HMA-FO ODA, June 2011 Slide 8

Deliverables - OGC Documents – OGC WCS_2. 0 Bundle (http: //www. opengeospatial. org/standards/wcs) WCS 2. 0 Core (OGC 09 -110 r 3) GML 3. 2. 1 Application Schema - Coverages (OGC 09 -146 r 1) WCS 2. 0 Extension: KVP protocol (OGC 09 -147 r 1) WCS 2. 0 Extension: XML/POST protocol (OGC 09 -148 r 1) WCS 2. 0 Extension: SOAP protocol (OGC 09 -149 r 1) …plus 20 XML schema & example files Date (submitted 2010 -11 -04) passed 2010 -08 -18) OGC 10 -140 OGC Web Coverage Service 2. 0 Interface Standard - Earth Observation Application Profile v. 0. 2. 0 (2011 -01 -19) 2011 -05 -06 (https: //portal. opengeospatial. org/files/? artifact_id=42579) WCS 2. 0 Test Suite is available, based on the OGC 09 -147 r 1 Abstract Test Suite (https: //svn. opengeospatial. org/ogcprojects/cite/scripts/wcs/2. 0. 0/) OGC 11 -010 WCS 2. 0 Extension for CF-net. CDF & OPe. NDAP OGC 10 -147 WCS 2. 0 Format Extension - Geo. TIFF v. 0. 0. 1 WCS 2. 0 Overview: Core & Extensions (Best Practice) (OGC 09 -153) HMA-FO ODA, June 2011 Status Adopted as Implementation Specification Approval by OGC OAB 2011 -05 -10 First version 2011 -03 -02 2010 -07 -06 Draft 2010 -11 -04 Slide 9 Draft

Developments - Software Status WCS 2. 0 extension to Map. Server 6. 0 (official release) 2011 -05 -12 EOx. Server v. 0. 1. 2 (EO-WCS) Software Bundle (download – http: //eoxserver. org) 2011 -05 -04 Web. Map. Viewer WCS 2. 0/EO-WCS enabled Client (GIM) 2011 -03 -17 Demonstration Server (WCS 2. 0 / EOx. Server) v. 0. 1. 2 at ESA (within a virtual machine) is online 2011 -05 -17 HMA-FO ODA, June 2011 Slide 10

Architecture 1/3 Written in Python Based on entirely on Open Source Software (Map. Server, Django, GDAL, Spatial. Lite or Post. GIS, PROJ. 4, etc. ) Map. Server has been extended to support WCS 2. 0 (see ticket at http: //trac. osgeo. org/mapserver/ticket/3595) § Extension is included in Map. Server 6. 0 Implements Model – View – Controller concept HMA-FO ODA, June 2011 Slide 11

Architecture 2/3 HMA-FO ODA, June 2011 Slide 12

Architecture 3/3 HMA-FO ODA, June 2011 Slide 13

Functionality 1/3 Implements EO-WCS and EO-WMS on top of Map. Server's 6. 0 WCS and WMS implementations Support of GML AP – Coverages for Rectified. Grid. Coverages Support of adopted WCS 2. 0 specification Anticipated support of envisaged extensions: Coverage format, Geo. TIFF encoding, predefined (or EPSG) CRSs, scaling & interpolation, and non-referenced access Protocol bindings supported § KVP (tested) § XML/POST (used together with future SOAP Proxy (currently developed in the O 3 S project) to support XML/SOAP protocol binding) HMA-FO ODA, June 2011 Slide 14

Functionality 2/3 Support of 2 -D EO Coverages derived from gmlcov: Rectified. Grid. Coverage Support of Dataset Series as a collection of EO Coverages e. g. in a time series Support of new Describe. EOCoverage. Set operation on Dataser Series and EO Coverages Support of Stitched Mosaic of Rectified EO Coverages including concept of contributing. Footprint Support of EO Metadata (retrieval and evaluation in Describe. EOCoverage. Set operation) HMA-FO ODA, June 2011 Slide 15

Functionality 3/3 Support of EO-WMS for EO Coverages Coverage formats supported § Geo. TIFF § Formats supported by the GDAL library (support needs to be verified for each required format separately) Extensive unit tests based on MERIS sample data documenting available functionality Documentation available based on Sphinx HMA-FO ODA, June 2011 Slide 16

WCS / EO-WCS Client Based on evolution of the Service Support Environment (SSE) Web. Map. Viewer WMS, WFS, WCS, SOS Client Implemented with a new technology baseline (JSR 286 Portlets, Java Server Faces) Combining thin browser based Javascript client with portrayal server and Content Management System Work performed in HMA-FO was to Support the WCS 2. 0 Core Specification Support the EO extension of WCS 2. 0 HMA-FO ODA, June 2011 Slide 17

WCS / EO-WCS Client Allows visualization of coverages, dataset series and stitched mosaics with functionality to Influence portrayal RGB/Grey scales Band mapping Image enhancement Transparency Discoverages inside dataset series Show coverage metadata Download coverages HMA-FO ODA, June 2011 Slide 18

EO-WCS Concepts - Overview Data structures: § EO Coverage = Coverage + EO Metadata + Lineage • Rectified. Grid. Coverage or • Referenceable. Grid. Coverage Dataset = 2 D “horizontal” EO Coverage § Stitched Mosaic = homogeneous grouping § • Can be seen as a coverage itself Dataset Series = heterogeneous grouping Operations: Capabilities document: more flexible Describe. Coverage, Get. Coverage: as known Describe. EOCoverage. Set: spatio-temporal search possibility § HMA-FO ODA, June 2011 Slide 19

Definitions EO-WCS EO vocabulary EO-WCS scene / image / product / raster data (image) / raw data (image) / orthoimage dataset mosaic / seamless mosaic google-like mosaic / composite mosaic stitched mosaic series / time series dataset series collection dataset series HMA-FO ODA, June 2011 Slide 20

Definitions EO-WCS Term Definition grid refers to data represented by values at each point in a "raster" coordinate system (as opposed to "vector" geodata) quadrilateral a polygon with 4 sides, usually a rectangle or square, but could also be a rhombus, kite, etc. rectified grid having regular spacing in a projected or geographic CRS referenceable grid not rectified, but associated with (one or more) coordinate transformations which relate the image or engineering CRS to a projected or geographic CRS HMA-FO ODA, June 2011 Slide 21

Definitions EO-WCS EO vocabulary EO-WCS scene / image / product / raster data (image) / raw data (image) / orthoimage dataset mosaic / seamless mosaic dataset lat long i rectified grid coverage j refereceable grid coverage HMA-FO ODA, June 2011 Slide 22

Definitions EO-WCS EO vocabulary series / time series EO-WCS dataset series Describe. EOCoverage. Set provides a set of dataset descriptions t lat long extraction from a 2 -D dataset series by repeated Get. Coverage Requests HMA-FO ODA, June 2011 Slide 23

Definitions EO-WCS EO vocabulary EO-WCS google-like mosaic / composite mosaic stitched mosaic lat t dataset series Describe. EOCoverage. Set (mosaic ID) provides a set of descriptions of the dataset used in the mosaic long 2 -D stitched mosaic Describe. Coverage (mosic ID) provides a descriptions of the mosaic lat HMA-FO ODA, June 2011 long Slide 24

Definitions EO-WCS EO vocabulary EO-WCS collection dataset series E A B t A …. . dataset (e. g. scene) B …. . dataset (e. g. seamless mosaic) C …. . datasets of dataset series (e. g. time series) D …. . datasets of dataset series E …. . dataset series (e. g. collection) C D lat A dataset may be a member in different dataset series long HMA-FO ODA, June 2011 Slide 25

Dataset Series - Past What needs to be done if you just need a small AOI ? Discover the required data using a Catalog Go to FTP-site Discover respective files Download 1 st big file Wait until download is finished Download 2 nd big file Wait … Download 3 rd big file Wait … Download other big files Wait … Extract the AOI from each file and throw away the rest HMA-FO ODA, June 2011

Dataset Series - Past With all that waiting, one may end up like this …. Credit: Ambro HMA-FO ODA, June 2011 Slide 27

Dataset Series - Now What can EOWCS do for you if you just need a small AOI ? You may discover the required data using a Catalog You may Order using direct links from the Catalog for each item OR … You may use a EO-WCS enabled Client § Request information about the dataset series § Supply the AOI § Supply your time-frame (TOI) § You may request details for each Dataset Request the datasets of your AOI/TOI and Wait until download of the AOIs requested is finished HMA-FO ODA, June 2011 Slide 28

Dataset Series - Now That's much more rewarding … Credit: Ambro HMA-FO ODA, June 2011 Slide 29

Stitched. Mosaic - Now EO-WCS allows to keep track of the metadata history of stitched mosaics? EO Metadata Record-1 But each pixel will keep its history EO Metadata Record-2 Each coverage has its metadata record Overlaying coverages replace existing ones HMA-FO ODA, June 2011 Slide 30

Improvements with WCS / EO-WCS Users § No more endless file-listings on FTP-Servers without knowing file-content § Less data download necessary you get only what you really need (AOIs/TOIs) § Access to original data Providers § Offer original data or rectified data § No data duplication required § Possibility of full integration into complex service chains (e. g. pass a coverage to a Web Processing Service, WPS) § Better quality of service provisioning HMA-FO ODA, June 2011 Slide 31

WCS & EOx. Server Future (1) Aim: EOx. Server to become official Reference Implementation of EO-WCS Full integration of results from OWS-8 § Support of the envisaged band subsetting extension § Coverage format support: • Net. CDF (including the option for CF-net. CDF) (via GDAL) • JPEG 2000 (via some available Open Source implementation (TBD)) • HDF-EOS (via GDAL) Ongoing development in parallel ESA funded project O 3 S (Online Open-standard Observation Service) Demonstrate scalability with large number of coverages ( FP 7 Earth. Server Project) Coupling of rasdaman and EOx. Server (combine WCPS and EO-WCS) HMA-FO ODA, June 2011 Slide 32

WCS & EOx. Server Future (2) Support FTP as Back-end access protocol (e. g. towards legacy Rolling Archives, Virtual Archives) Demonstrate suitability for complex Use Cases WCS-T, WCPS, Referenceable. Grid. Coverages WCS extensions (Standards & Implementation): CRS, Scaling & Interpolation, Range. Subsetting Tools for automation of Administration tasks (e. g. hourly updating of offered coverages) Provision of TEAM 2. 0 compatible conformance test suites Generalize EOx. Server to support other Map-Servers Update WCS Client to support further extensions HMA-FO ODA, June 2011 Slide 33

Summary Task-3 Achievements WCS 2. 0 released as OGC Implementation Standard EO-WCS as OGC Implementation Standard (to be released soon) Implementation of WCS 2. 0 Server ( Map. Server) Implementation of EO-WCS Server ( EOx. Server) Implementation of WCS 2. 0/EO-WCS enabled Client ( Web. Map. Viewer ) HMA-FO ODA, June 2011 Slide 34

Demonstration – Server Administration Interface – show easiness of adding a Dataset Series to EOx. Server (3 min) Demonstration – Client Integrated in overall Demo-Scenario § Discover EO Product within a Dataset Series through AOI and time period (TOI) § § Visualize Coverage Influence Portrayal Download Coverage Show coverage metadata HMA-FO ODA, June 2011 Slide 35

Contacts EOX IT Services Gmb. H, • stephan. meissl@eox. at • christian. schiller@eox. at Austria (Prime) G. I. M. Geographic Information Management nv/sa, Belgium • steven. smolders@gim. be Jacobs University Bremen g. Gmb. H, • p. baumann@jacobs-university. de Spot Image SA, France • philippe. merigot@spotimage. fr HMA-FO ODA, June 2011 Germany