HMA for Science SRR Download Protocol SRR 16

HMA for Science SRR Download Protocol SRR 16 May 2013, ESRIN Frascati Daniele Marchionni, Telespazio HMA-S SRR, 16 May 2013 Slide 1

Download Protocol - Summary Ø Status of the Document Ø Protocol Overview Ø Requirements Classes Ø Conformance Classes and tests HMA-S SRR, 16 May 2013 Slide 2

Status of the document Ø OGC 13 -042 OGC RESTful Encoding of Ordering Services Framework For Earth Observation Products (ROSEO) V 0. 9. 1 2013/05/14 Ø Single zip file delivered including: • The document: 13042_RESTful_Encoding_Ordering_Services_for_Earth_Observati on_Products. docx • The schema file: § roseo. xsd • Many examples XML files: § ROSEO generic messages; § Sentinel-1, Sentinel-2, Sentinel-3 order options examples HMA-S SRR, 16 May 2013 Slide 3

Status of the document Ø The document has been prepared following the latest template from OGC (10176 r 4_New_OGC_Document_Template_Draft) Ø Requirement Classes provided in sections § 8 (Core), § 9 (Scene Selection), § 10 (Subscription. Order) Ø Conformance Classes & tests provided in section § 11 Ø The OGC doc ID has been reserved (13 -042), but document not uploaded yet because waiting for SRR comments before posting HMA-S SRR, 16 May 2013 Slide 4

ROSEO Overview Ø The OGC 13 -042 OGC RESTful Encoding of Ordering Services Framework For Earth Observation Products (ROSEO) proposes as OGC Best Practice document the simplified “RESTful” version of the HMA Ordering ICD (OGC 06141 r 6) allowing accessing to Ordering functions via appropriate HTTP requests and not via SOAP calls. Ø In this way Ordering functions will be available to Web browsers without the need of more complex infrastructures based on SOAP. Ø This protocol covers, in RESTful way, the following HMA Ordering ICD operations: • Get. Capabilities • Get. Options • Submit • Get. Status • Describe. Result. Access • Cancel HMA-S SRR, 16 May 2013 Slide 5

ROSEO Overview Ø ROSEO Document approach: • Structured by Resources: Capabilities, Order. Options, Ordered. Product. Each resource has a dedicated section including description, URLs, supported HTTP methods. • Addressability: each identified resource has specified URLs. • Uniform interface: the operations performed on resources are only the Methods of HTTP protocol: § GET for reading a resource; § POST for creating a resource with ID defined by the server; § PUT for creating / replacing a resource with ID defined by the client; § DELETE for deleting a resource. • Connectdness: each resource has links (URLs) to the other related resources. HMA-S SRR, 16 May 2013 Slide 6

ROSEO Overview Ø Simplifications with respect to OSEO: • Removed order of products derived from a programming request. • Removed support for order quotation, function very rarely used from operational implementations of OSEO specification. • Removed support for asynchronous notifications: in fact this document focuses on Web Applications in general and on REST approach in particular, while the asynchronous message exchange is more a SOAP Web Service Oriented function. • Simplified delivery mechanism: only direct download from URL (HTTP GET) is supported: delivery via media and via FTP not included. HMA-S SRR, 16 May 2013 Slide 7

ROSEO Overview Ø Summary of changes with respect to OSEO: • Capabilities: § Removed support for notifications; § Added resource URLs in Contents section (connectdness); • Order. Options: § Changed root element removing the reference to the operation (Get. Options. Response) and putting just Order. Options; § Added URL for linking to OSDD / direct download of the product; • Order: § Changed root element removing the reference to the operation (Submit. Response / Get. Status. Response) and putting just Order; § Merged definitions for Order submit and Order monitoring Order; § Added URLs for getting directly the ordered products (sort of Describe. Result. Access call); § Added URLs for getting directly the order or the order items. • Ordered. Product: § Very minor changes with respect to OSEO: Describe. Result. Access. Response HMA-S SRR, 16 May 2013 Slide 8

ROSEO Resources – Capabilities - schema Ø Capabilities resource contains metadata about the ROSEO server reporting e. g. : service identifier, description, restrictions, service provider, supported operations, etc. Ø ROSEO reuses the Capabilities definition provided by OSEO (OGC 06 -141 r 6) applying a profiling for setting values compatible with ROSEO purposes. HMA-S SRR, 16 May 2013 Slide 9

ROSEO Resources – Capabilities - URLs Ø URLs <Service Root URL> http: //<hostname>: <port>/<context path> <ROSEO Root Path> <Service Root URL>/ROSEO/1. 0. 0 URL <Capabilities Resource URL> <Service. Identification section> <Service. Provider section> <Contents section> Definition <ROSEO Root Path>/Service. Identification <ROSEO Root Path>/Service. Provider <ROSEO Root Path>/Contents HMA-S SRR, 16 May 2013 Slide 10

ROSEO Resources – Capabilities – HTTP Methods Ø HTTP Metods HTTP Method GET URL Definition <ROSEO Root Path> Request Entity Response Entity Body Empty <roseo: Capabiliti es> element instance POST It returns the complete Capabilities document. <ROSEO Root It returns the Empty Path>/Service. Identification element <ROSEO Root It returns the Empty Path>/Service. Provider element <ROSEO Root It returns the Empty Path>/Contents element Not used by this document. PUT Not used by this document. DELETE Not used by this document. HMA-S SRR, 16 May 2013 <ows: Service. Iden tification> element instance <ows: Service. Prov ider> element instance <roseo: Contents> element instance Slide 11

ROSEO Resources - Order. Options Ø Order Options resource describes the parameters that can be set by the clients for customizing an item to order. Ø Depending on the Satellite mission, order options can be defined either at collection level, which means that all EO Products belonging to a specified collection have the same order options, or at product level, in this case each product has potentially different order options. Ø Two Order Options formats have been defined: • The “canonical” XML format, i. e. the XML format directly derived from OSEO; • the OSDD one, which reports the order options via template URL plus XML block in “canonical” format. HMA-S SRR, 16 May 2013 Slide 12

ROSEO Resources – Order. Options – XML format Ø Almost identical to OSEO: order option groups; SWE Common; processing, delivery and scene selection options. Ø Differences: • renamed tags: Order. Options, Order. Option. Group • Added download. URL element HMA-S SRR, 16 May 2013 Slide 13

ROSEO Resources – Order. Options – OSDD format Ø Order Options in OSDD format includes: • Extension to the OSDD, via the foreign markup mechanism, for including Order. Options resource in “canonical” XML format. As specified in Open. Search standard, “Open. Search Description Documents can be extended with new XML “foreign” elements and attributes provided that all of them are associated with an explicit XML namespace. Clients that encounter unrecognized foreign elements should ignore them and continue to process the document as if these elements did not appear”. • Extension to the Open Search Template URL for including Order. Options parameters as template parameters. • The namespace of these extensions is the ROSEO one: § Prefix: roseo § URL: http: //www. opengis. net/roseo/1. 0 HMA-S SRR, 16 May 2013 Slide 14

ROSEO Resources – Order. Options – OSDD example <? xml version="1. 0" encoding="UTF-8"? > <mstns: Open. Search. Description xmlns: oseo="http: //www. opengis. net/oseo/1. 0" xmlns: roseo="http: //www. opengis. net/roseo/1. 0" xmlns: swe="http: //www. opengis. net/swe/2. 0" xmlns: xlink="http: //www. w 3. org/1999/xlink" xmlns: mstns="http: //a 9. com/-/spec/opensearch/1. 1/" xmlns: xsi="http: //www. w 3. org/2001/XMLSchema-instance"> <mstns: Short. Name>Order Options</mstns: Short. Name> <mstns: Description>OSDD for Order Options</mstns: Description> <mstns: Url type="application/atom+xml" template="http: //some. eocat. server/ESA: EECF: ENVISAT_ASA_IMx_x. S? q={search. Term}& count={count}& start. Index={start. Index? }& st art. Page={start. Page? }& language={language? }& input. Encoding={input. Encoding? }& output. Encoding={output. Encoding? }& bbox={ge o: box? }& geom={geo: geometry? }& id={geo: uid? }& lat={geo: lat? }& lon={geo: lon? }& radius={geo: radius? }& rel={geo: relation ? }& loc={geo: name& }& startdate={time: start? }& stopdate={time: end? }& prclv={roseo: Processing. Level}& prd t={roseo: Product. Type}& qos={roseo: Quality. Of. Service}"/> <mstns: Syndication. Right>open</mstns: Syndication. Right> <mstns: Adult. Content>false</mstns: Adult. Content> <mstns: Language>*</mstns: Language> <mstns: Input. Encoding>UTF-8</mstns: Input. Encoding> <mstns: Output. Encoding>UTF-8</mstns: Output. Encoding> <roseo: Order. Options xsi: schema. Location="http: //www. opengis. net/roseo/1. 0 roseo. xsd" xmlns: oseo="http: //www. opengis. net/oseo/1. 0" xmlns: roseo="http: //www. opengis. net/roseo/1. 0" xmlns: swe="http: //www. opengis. net/swe/2. 0" xmlns: xlink="http: //www. w 3. org/1999/xlink" xmlns: xsi="http: //www. w 3. org/2001/XMLSchema -instance"> <roseo: Order. Option. Group> <oseo: product. Order. Options. Id>Level 1, Product PRI (ASA_IMP)</oseo: product. Order. Options. Id> <oseo: description>Order Options for producing ASAR IMP 1 P product</oseo: description> <oseo: order. Type>PRODUCT_ORDER</oseo: order. Type> <oseo: option> <swe: Data. Record> <swe: field name="processing. Level"> <swe: Category updatable="false" optional="true" definition="http: //www. opengis. net/def/parameter/ESA/1. 0/processing. Level"> <swe: identifier>processing Level</swe: identifier> <swe: constraint> <swe: Allowed. Tokens> <swe: value>1 B</swe: value> </swe: Allowed. Tokens> HMA-S SRR, 16 May 2013 </swe: constraint> Slide 15 </swe: Category>

ROSEO Resources – Order. Options - OSDD Ø Order. Options declaration (when retrieving options): it is a template parameter in a URL. Example: http: //roseo. server. com/<path>? <other parameters>& prclv={roseo: Processing. Level} Ø Order. Options set (when creating the request): • in case of simple order options i. e. defined using basic types like xs: int, xs: string, xs: date. Time, the setting is straightforward: just replace the Template parameter with the URL encoded value. Example: http: //roseo. server. com/<path>? <other parameters>& prclv=1 B&prdt=ASA_IMS_1 P&qos=STANDARD • in case of structured order options i. e. defined using complex types like: arrays, structures and nested structures the mapping is more complex because a rule has to be defined for encoding the values of subparameters. For the sake of simplicity the well known and vastly used JSON syntax [NR 16] has been adopted. Example: http: //roseo. server. com/<path>? <other parameters>&sbcc={“Red. Band”: “band 1”, “Green. Band”: “band 6”, “Blue. Band”: “band 12”} HMA-S SRR, 16 May 2013 Slide 16

ROSEO Resources – Order. Options – URL & Methods HTTP Method GET URL Request Entity Response Entity Body <ROSEO Root It returns the Order Empty <roseo: Order. Options> Path>/options/{ID}/{format} Options related to element instance. the specified item (product / collection / subscription). POST PUT DELETE Not used by this document. Names {ID} {format} Definition ID of the resource to get Order. Options: either an EO Product or a collection or a subscription. Required Order. Options format: either canonical XML format or OSDD format HMA-S SRR, 16 May 2013 Data type and value xs: string (max 255 chars) URL encoded xs: string (max 255 chars) Allowed Values: xml osdd Slide 17

ROSEO Resources - Order Ø Order resource is the envelope sent from the client to order a set of EO products with the selected order options or for subscribing to an existing EO Product subscription. Additionally the Order resource includes also status information reporting the current status of the order processed by the server. Ø This resource can be accessed as a whole, including the whole order, or in parts by accessing to the various ordered items. HMA-S SRR, 16 May 2013 Slide 18

ROSEO Resources - Order HMA-S SRR, 16 May 2013 Slide 19

ROSEO Resources - Order HTTP Method GET POST PUT DELETE URL <ROSEO Root Path>/order/{order identifier} Definition Request Entity Response Entity Body It returns the Order related Empty <roseo: Order> to the specified order element instance. identifier. It returns the Order Item Empty <roseo: order. Item> within the specified Order element instance. <ROSEO Root Path>/order/{order identifier}/{order item identifier} <ROSEO Root It returns all orders Path>/order? {filter expression} matching the filter expression. <ROSEO Root Path>/order It allows to submit an order to the ROSEO server. The ID of the submitted order is returned in the Response Entity Body. Not used by this document <ROSEO Root To flag as Cancelled the Path>/order/{order identifier} specified Order resource. The Order resource is still on the server. HMA-S SRR, 16 May 2013 Empty <roseo: Orders> element instance. <roseo: Order> element instance. <oseo: order. Id> with the URL of the created order resource. Empty Slide 20

ROSEO Resources - Ordered. Product Ø This resource allows to physically retrieve the ordered products once they are ready for download (almost identical to OSEO: Describe. Result. Acccess operation) HMA-S SRR, 16 May 2013 Slide 21

ROSEO Resources - Ordered. Product HTTP Method GET POST PUT DELETE URL Request Entity Body It returns all Ordered Empty Product resources related to the specified order. Response Entity Body <roseo: Ordered. Pr oducts> element instance. <ROSEO Root Path>/order/{order identifier}/{order item identifier}/file It returns the Ordered Product resource related to the specified order item. Empty <roseo: Ordered. Pr oducts> element instance. <ROSEO Root Path>/order/{order identifier}/file? {filter expression} It returns all Ordered Empty Product resources related to the specified order matching the specified filter. <roseo: Ordered. Pr oducts> element instance. <ROSEO Root Path>/order/{order identifier}/file Definition Not used by this document HMA-S SRR, 16 May 2013 Slide 22

ROSEO Error Handling Ø In the event that a ROSEO server encounters an error during the access to some resource, then it returns an HTTP response including: • HTTP Status Code: 4 XX for errors on the client side; 5 XX for errors on server side. • HTTP Entity Body: ows: Exception. Report element set as specified in § 8 of [NR 7]. Ø The specific values for Status Code and ows: Exception. Report depend on the accessed resource and performed HTTP method. There are dedicated errors for each of them. HMA-S SRR, 16 May 2013 Slide 23

ROSEO Order. Options – Sentinel 1 Ø Sentinel-1 products are structured by “Native Datasets” (i. e. Collections): each of them has EO Products with specific Metadata Attributes and “Download Options” (i. e. Order. Options). Ø The possible Download Options are: • Processing, enumerated string • Resolution, enumerated string • Swath, enumerated string • Safe, boolean HMA-S SRR, 16 May 2013 Slide 24

ROSEO Order. Options – Sentinel 1 Ø The important point to note is that the values of the different options are correlated e. g. : • Resolution can be specified only when processing is GRD; • Swaths EW<N> can be specified only for some datasets, while IW<N> can be specified for other datasets • Etc. Ø All of these dependencies can be represented via dedicated Order. Option. Group: each group specifies the possible order options with the correlated allowed values. Once a client has selected a group, then only the corresponding order options and values are returned. Ø Then no special issues. HMA-S SRR, 16 May 2013 Slide 25

ROSEO Order. Options – Sentinel 2 Ø Sentinel-2 products are structured by “Native Datasets” (i. e. Collections): each of them has EO Products with specific Metadata Attributes and “Download Options” (i. e. Order. Options). Ø The possible Download Options are: • Band (ROSEO: Spectral. Band. Composition), TBD values. • pvi, boolean option • aux. Data, boolean option • metadata. Level, enumerated string • merging. Flag, boolean option. • output. Format (OSEO: Format), enumerated string HMA-S SRR, 16 May 2013 Slide 26

ROSEO Order. Options – Sentinel 2 Ø To be noted that: • All datasets have the same options; • One option does not depend on the values of the other ones; • Sentinel-2 has very specific order options which cannot be made as standard (e. g. pvi, aux. Data, metadata. Level, merging. Flag). • Then Sentinel-2 order options are much easier than Sentinel-1: a single combination of order options is good for all datasets. HMA-S SRR, 16 May 2013 Slide 27

ROSEO Order. Options – Sentinel 3 Ø Sentinel-3 does not support any processing, but the products are delivered as they are so there are not processing options. Nevertheless it support very fine grained sub-setting options i. e. it allows to specify, of the whole bunch of files composing the product only those of interest. This capability is provided via the following “Download Options”: • mdf – Measurement Data Files download options: which allows to select either the whole file or only the MDFs of interest. Multiple selections are also possible. • adf – Annotation Data Files download options: which allows to select either all annotations or only the annotations of interest. Multiple selections are possible. Ø Also mdf and adf are often linked each other: i. e. when a mdf is selected, then only some adf are possible. HMA-S SRR, 16 May 2013 Slide 28

ROSEO Order. Options – Sentinel 3 Ø Ø The following cases have been noticed: • In case of no options or independent options or options made independent then no problem, just define two order options: mdf and adf (both swe: Category) with some predefined values. • In case of very linked options having several groups of mdf values linked with corresponding group of adf values, then the representation is more complex: § In fact SWECommon allows to model restrictions on parameters, but it is supported only for simple values (swe: Category, swe: Count, swe: Quantity, etc. . ) and not for structured types like swe: Date. Record which instead would be necessary for representing a pair of attributes. § Also we cannot use the Order. Option. Group for representing correlated values because it should be possible to perform multiple selections, while Order. Option. Group elements are mutually exclusive. § The envisaged solution is to put mdf and adf within a single parameter mdf_adf further structured via swe: Data. Choice putting in a single choice the group of mdf and adf linked values. In conclusion: • swe: Data. Array has been used for allowing multiple selections; • swe: Data. Choice for regrouping the correlated pair of options; • swe: Data. Record for defining the pair mdf & adf. HMA-S SRR, 16 May 2013 Slide 29

ROSEO Requirement Classes Ø Ø The document is a Best Practice, so it includes: • Requirements Classes • Conformance Classes and Conformance Tests (ATS) Requirement Classes: HMA Ordering ICD requirement classes have been reused and simplified: in fact only 3 out of 17 classes have been defined / reused: • Core, which specifies the minimum behaviour that all ROSEO servers shall implement. It includes the support of ordering of precisely identified Earth Observation Products. • Scene. Selection, which specifies the additional requirements a ROSEO server has to comply for supporting scene selection. • Subscription. Order, which specifies the additional requirements a ROSEO server shall implement for supporting subscription orders. HMA-S SRR, 16 May 2013 Slide 30

ROSEO Conformance Classes Ø Ø Conformance Classes: normally there is a 1: 1 mapping between Requirement Classes and Conformance Classes, so we have: • Core Conformance Class: a server complying with this class provides support for all ROSEO resources and operations, but limited to ordering of precisely identified products. • Scene. Selection Conformance Class: with respect to Core class it adds the support for scene selection extraction from requested products. • Subscription. Order Conformance Class: a server complying with this class provides full support for automatic download of subscribed products. Conformance Tests: A conformance class is implemented via a set of Conformance Tests, each testing one or more requirements of the corresponding Requirements Class. HMA-S SRR, 16 May 2013 Slide 31

ROSEO Requirements Statistics Class Resource Requirements Total Core Capabilities 15 Core Order. Options 12 Core Order 20 Core Ordered. Product 7 TOTAL 54 Scene. Selection Capabilities 0 Scene. Selection Order. Options 1 Scene. Selection Order 3 Scene. Selection Ordered. Product 0 TOTAL 4 Subscription. Order Capabilities 1 Subscription. Order. Options 2 Subscription. Ordered. Product 2 TOTAL 7 HMA-S SRR, 16 May 2013 Slide 32

ROSEO Conformance Tests Statistics Class Resource Requirements Total Core Capabilities 5 Core Order. Options 5 Core Order 12 Core Ordered. Product 4 TOTAL 26 Scene. Selection Capabilities 0 Scene. Selection Order. Options 1 Scene. Selection Order 3 Scene. Selection Ordered. Product 0 TOTAL 4 Subscription. Order Capabilities 1 Subscription. Order. Options 1 Subscription. Order 2 Subscription. Ordered. Product 1 TOTAL 5 HMA-S SRR, 16 May 2013 Slide 33