Grid and the ESA Technology Programme Pier Giorgio

Grid and the ESA Technology Programme Pier Giorgio Marchetti EOP/GDR Research & Technology Development Section Pier. Giorgio. Marchetti@esa. int ESA Grid Technology ESA, 1

GREASE Project Status • First delivery July 2003 • CCN-02: – Evaluate GREASE with large scale production of test data – Evaluate use of Globus Toolkit 3 resources and XMLSpy • Timeframe: September 2003 - December 2003 • Currently working on the large scale production of test data for the OMI instrument for the EOS-AURA mission. ESA Grid Technology ESA, 2

• Data processing applications of the Ozone Monitoring Instrument (OMI): – Simulator – L 0 processor – L 1 B processor ESA Grid Technology ESA, 3

Simulation and Validation of EO Instruments Environment Orbit Model Atmosphere Model Instrument Model Downlink Model L 0 Processor L 1 b Processor L 2 Processor Information ESA Grid Technology ESA, 4

OMI Validation Spectrum generator Instrument simulator Level 1 b processor Level 22 Level 2 processor compare spectra compare atmospheric parameters • Simulator and Level 1 b Processor are being developed by Dutch Space (the Netherlands). Level 2 Processors are being developed at KNMI (the Netherlands), FMI (Finland) and NASA (United States). • Simulation + Level 1 b Processing takes about 15 minutes on a Pentium III for a single ground pixel. ESA Grid Technology ESA, 5

OMI Level 0 Data Generation • Purpose: Test of the OMI Ground Segment – Level 1 B processing in Science Investigator Processing System (SIPS) – Archiving in Goddard Distributed Active Archive Center (DAAC) – Higher level processing in SIPS and OMI Data Processing System (ODPS) • Data set should represent at least 1 month of OMI Level 0 data. • Data set should contain representative science data. ESA Grid Technology ESA, 6

Ozone Monitoring Instrument (OMI) • Imaging spectrograph – wavelength range: 270 - 500 nm (UV and Visible) – high resolution: 24 x 14 km – 60 ground pixels per measurement (2 seconds) – 1500 measurements per orbit ESA Grid Technology ESA, 7

OMI Instrument Response Simulator • Used for: – testing of Level 1 b algorithms and data processor – calculation of instrument performance parameters – development of instrument measurement types • Single measurement per run – 90 - 900 sec. execution time per run (depending on simulation mode) – generates science and engineering data in HDF format • Limitation on number of input spectrums – 1 or 2 spectrums for 60 ground pixels – Not possible to use separate spectrums for each ground pixel • Runs on UNIX, Linux and Windows ESA Grid Technology ESA, 8

Using OMI Simulator for Data Generation • Simulate each measurement during a one month period – 420 orbits, 1. 500 measurements per orbit - 630. 000 measurements – 630. 000 simulations, 90 sec. per simulation - 15. 750 hours (656 days) of processing!! • All simulations can be done independent, so multiple resources can be used in parallel. • Pre-processing: input spectrums are needed for the simulation. • Post-processing: output data must be converted from HDF to Level 0 Data Sets. ESA Grid Technology ESA, 9

Using GOME Data as input for OMI Simulator • Spectrums can be extracted from existing GOME Level 1 data (available in the EU Data. Grid project) • Ground tracks must be mapped – OMI: 60 ground pixels (24 x 14 km) every 2 seconds – GOME: 3 ground pixels (320 x 40 km) every 6 seconds • Spectrum data can be extracted from 2 GOME ground pixels and used for 3 OMI simulation runs • GOME orbit data is available ESA Grid Technology ESA, 10

One Month In the Live of OMI on EOS-AURA • Using GOME measured data (provided by KNMI) Measured Data Spectrum Extractor OMI Response Simulator L 0 Data Generator OMI L 0 data • Number of times each of the applications must be executed Measured Data Spectrum Generator OMI Response Simulator L 0 Data Generator OMI L 0 data 2 x 1400 x 420 x ESA Grid Technology ESA, 11

The Complete Workflow ESA Grid Technology ESA, 12

Architecture (1) • Three major elements in GREASE: ESA Grid Technology ESA, 13

Architecture (2) Registry • Application information – general information – interface information • Application deployment – static (bound to a single node) – dynamic (moved to other suitable nodes) • Resource information (computation and storage) – now: static – future: dynamic (e. g. Globus MDS, EDG Information System, . . . ) • Resource type – now: Globus Toolkit 2 – future: other resource types (e. g. Globus Toolkit 3, Unicore, . . . ) ESA Grid Technology ESA, 14

Architecture (3) Workflow Engine • Main functions now: – receives complex jobs in the form of workflows – selects Grid nodes for execution – handles certificates – submits applications – handles file transfer between applications – provides information for monitoring – Globus Toolkit 2 environments only (via Globus Co. G-kit) • Future – other Grid environments (e. g. Globus Toolkit 3, Unicore, . . . ) – other forms of interprocess communication (e. g. MPI, . . . ) ESA Grid Technology ESA, 15

Architecture (4) User Tools • Create and submit workflows • Monitor progress of running jobs • Manage information in the Registry • Have Graphical User Interfaces • Written in Java – runs on Windows, Linux, UNIX. . . ESA Grid Technology ESA, 16

Architecture (5) Communication • SOAP via HTTP • firewall friendly • Grid middleware specific – Globus Toolkit 2 (GRAM, GASS) ESA Grid Technology ESA, 17

GREASE Controller (2) • Apache Web Server + Apace Axis software handle SOAP calls. • Registry and Job information stored in My. SQL database. Globus specific protocols SOAP My. SQL Database Apache AXIS Apache Jakarta Tomcat Web Server JDBC Connector Workflow Tool Apache AXIS GREASE Workflow Engine User Workstation Data Processing Application Java Co. G-kit Controller ESA Grid Technology Globus Toolkit Grid Resource ESA, 18

Intuitive graphical user interface ESA Grid Technology ESA, 19

How does it work? Application Data Grid cluster Application 1. Make workflow 2. Submit workflow 3. Execute tasks The Internet 4. Monitor progress 5. Transfer results Grid cluster Firewall Workflow tool Controller Intranet Scientist ESA Grid Technology ESA, 20

GREASE/Grid. Assist Test Grid Environment ESA Grid Technology ESA, 21

Grid. Assist Registry ESA Grid Technology ESA, 22

Grid. Assist Tests ESA Grid Technology ESA, 23

ENVISAT • 10 instruments on board • 200 Mbps data rate to ground • 400 Tbytes data archived/year • ~100 “standard” products • 10+ dedicated facilities in Europe • ~700 approved science user projects ESA Grid Technology ESA, 24

EOS-AURA OMI Status • Integration completed • TB/TV Tests ongoing • Launch March 2004 ESA Grid Technology ESA, 25

From GREASE to Grid Assist ESA Grid Technology ESA, 26

Why MASS ? - EO Value Chain Service DISTRIB. / DATA PROVIDER VALUE ADDER SP Info Data support / enabling system? USER Service Fusion / Manipulation Info Data PROVISION of… REQUEST for… SERVICE PROVIDER Other Domains Data Service: operational, repeated, regulated (usually by means of a Service Level Agreement) and agreed (via contract or agreement) exploitation for commercial or non-commercial purposes of an application. ESA Grid Technology ESA, 27

Objectives • Service support environment for the integration of Earth Observation and GIS services • Portal(s) for data providers, service providers and endusers • B 2 B technological demonstrator • Service integration ranging from basic to end-to-end Services • Open architecture, allowing to add services ESA Grid Technology ESA, 28

Objectives (cont’d) Global MASS Portal MASS User Global subnetwork Regional MASS Portal Access Point Regional or thematic subnetwork Local subnetwork MASS ICD Service Provider GIS Service Provider ESA Grid Technology EO Service Provider ESA, EO Data Provider 29

MASS Concepts • Multiple user types: – anonymous user: can only order free services • access via the MASS Portal – registered user: can order paying services • access via the MASS Portal – service provider: can add new services • access via the MASS Portal • access via workflow client tools – helpdesk, administrator • access via the MASS Admin Portal ESA Grid Technology ESA, 30

MASS Concepts (cont’d) • Service Types: – Function: executed locally on the MASS server, e. g. AOI selection, . . . – Basic service: executed on service provider server and published with a SOAP interface, e. g. “service A 1, A 2, …” I. e. Risk index, NDVI, Land Surface Temperature, etc. • may be called directly by user ordering it • may be called by workflow engine – (End-to-End) Service: chaining of basic services on MASS server, e. g. “service A”, i. e. overall risk index processor. ESA Grid Technology ESA, 31

MASS Concepts (cont’d) MASS Portal Users Workflow Net Primary Product A 1 (*) NPP or Net Primary Productivity represents repeatable estimates of the net flux of carbon. A 2 Service A 1: NPP processor (VITO) ESA Grid Technology Service A 3: Clipping (GIM) A 3 Service A 2: Format Conversion (GIM) ESA, Spot Végétation S 10 32

MASS Design • Main functions – user registration (anonymous, registered, service provider) – service registration – service directory with RFQ and Ordering – standard module for Area of Interest selection – module for service result visualisation on WMS layers – service chaining (workflow) – service subscription – news item management – link to external accounting/billing via XML file export. ESA Grid Technology ESA, 33

MASS Portal • J 2 EE application server – BEA Web. Logic Server 6. 1 / JBoss 3. 2. 2 – scalable over n computers in transparent way. • Workflow Support – BEA Web. Logic Integration 2. 1 – Collaxa Orchestration Server 2. 0 (BPEL) – workflow definition tool – workflow engine for service chaining etc. ESA Grid Technology ESA, 34

MASS Design (cont’d) • SOAP interfaces of “services”: – Catalogue Search (planned) • Earthnet On-line XML Front End ICD. – Catalogue Present (planned) • Earthnet On-line XML Front End ICD. – Request for Quotation (RFQ) • service provider returns price, availability etc. – Order • service provider returns result or “URL” of result • service provider returns 1 or more (subscription) order results ESA Grid Technology ESA, 35

MASS Design (cont’d) • Operations for Chained Services: – RFQ A = RFQ A 1 + RFQ A 2 + … (workflow) – Order A = Order A 1 + Order A 2 + … (workflow) – Search C = Search C 1 + Search C 2 + … (workflow) – Present C = Present C 1 or Present C 2 or … (workflow) • Interaction models: – asynchronous: operation results pass via MASS database. • Polling mechanism (obsolete) • Pushing mechanism (adoption of Ws-addressing) – synchronous: operation results immediately on same Web page (planned). ESA Grid Technology ESA, 36

MASS Design (cont’d) • Data (i. e. service result) distribution: – data included in SOAP message, textual or graphical (SVG) – data via FTP (URL in SOAP message) – data via HTTP (URL in SOAP message) – vector data via OGC WFS server (planned) – raster data via OGC WCS server (planned) • An example with FTP. . . ESA Grid Technology ESA, 37

Chaining Example Order A MASS User (8) (1) Result A : control flow : data flow : XML MASS Portal & Workflow Engine Internet (4) User FTP Server (5) Result A 1 Order A 1 Result A 2 (2) Order A 2 (7) (3) (6) Service Provider A 1 (e. g. VITO) (9) ESA Grid Technology Service Provider A 2 (e. g. GIM) ESA, 38

MASS Design (cont’d) • Combining services Fire stations Roads Community boundaries NDVI Risk Map Fire risk map (TPZ) LST Relief Chaining Overlaying – Is additional way to combine services. – Useful for vulnerability maps, hazard maps, rescue resources map, crisis map etc. (See GMES-TEASE study). ESA Grid Technology ESA, 39

MASS Design (cont’d) • MASS Toolbox – connect legacy systems and basic service deployment • MASS Portal – service registration – configuration of AOI tool • Workflow Editor – chained service creation • Workflow Console – workflow deployment – workflow monitoring ESA Grid Technology ESA, 40

Uses WSDL files of NPP and clipping services Chain of NPP and clipping ESA Grid Technology ESA, 41

Tracking based on Order ID Web-based monitoring of VITO service instances ESA Grid Technology ESA, 42

Web-based “auditing” of VITO service instance history ESA Grid Technology ESA, 43

MASS Protocols • BPEL (OASIS) – Business Process Execution Language • WSDL (W 3 C) – Models “web services” having SOAP bindings – Defines MASS and EO/GIS service interfaces – Extended according to Ws-addressing (for asynchronous) • SOAP (W 3 C) – message-based – Over HTTP, firewall-friendly, platform neutral – Optional use of HTTPS / SSL ESA Grid Technology ESA, 44

MASS Protocols (cont’d) • XML Schema (W 3 C) – RFQ and Order results are XML documents – Allows for stricter typing than DTD, e. g. ranges, dates, … – Service provider can define his own service “types” • XSLT (W 3 C) – stylesheet for service input/output presentation • XPath (W 3 C) – Defines “expression” notation allowing to extract an element from an XML (DOM) object – Used to extract e. g. “price” from RFQ result in workflow … ESA Grid Technology ESA, 45

MASS Protocols (cont’d) • ws-inspection (IBM, Microsoft) – distributed model for exposing WSDL descriptions – stored on MASS Portal itself – service discovery by search engines – alternative for UDDI • GML (Open. GIS) – Geography Markup Language – Used for Area of Interest data in SOAP messages – Used for service result viewing (e. g. oil spill, ship detection) – Used for catalogue footprint visualisation ESA Grid Technology ESA, 46

MASS Protocols (cont’d) • WMS (Open. GIS) – used for background of AOI selection – used for background layers of service result viewer • Other – OGC WFS: vector data distribution (planned) – OGC WCS: raster data distribution (planned) – OGC gazetteer: directories of cities and towns (planned) ESA Grid Technology ESA, 47

MASS Services (cont’d) AOI selection (GIM) NPP* Service VITO NPP input + AOI description + required format Format Conversion Service GIM NPP-Europe -HDF Clipping Service GIM NPP-Europe -Geo. TIFF NPP-Belgium -Geo. TIFF 1 (*) NPP or Net Primary Productivity represents repeatable estimates of the net flux of carbon. Product is based on Végétation S 10. ESA Grid Technology ESA, 48

Each service may have AOI Selection (cont’d) fixed order parameters, e. g. AOI, and service specific parameters, presented using a service specific stylesheet. Multiple methods to select AOI ESA Grid Technology 49

VITO NPP Service result presented with service XSLT stylesheet ESA Grid Technology ESA, 50

from persistent services… mass. wsdl mass. xsd … to transient web services MASS Portal ESA Grid Technology service. wsdl” service. xsd OGSA I/F ESA, GRID Service Backend 51

HICOD 2000 ESA Grid Technology ESA, 52

ESA Plan on GRID related Studies and Technology Developments • Concurrent Design Facility • Teletesting and Telemonitoring • THE VOICE 4 Q 2003 700 K GSP • GEO 3 Q 2004 500 K GSTP • Technology Research Programme • Further GSTP input & plans being defined ESA Grid Technology ESA, 53

For further information… • MASS "Yves Coene" yce@spacebel. be • GREASE “Mark ter Linden” m. ter. linden@dutchspace. nl • HICOD 2000 Miguel Dias” miguel. dias@iscte. pt ESA Grid Technology ESA, 54