ANABAS Use of Grids in Do D Applications

ANABAS Use of Grids in Do. D Applications Geoffrey Fox, Alex Ho, Marlon Pierce SBIR Briefing August 25, 2005 August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu

What do Web Services Prescribe? • The specify interfaces for system services (and generally useful services like database) • They specify an interface language (WSDL) for all services • They develop containers and frameworks to use to host services • They specify a message format (SOAP) for ALL messages that defines both application and system actions precisely • They imply a process be started to define domain specific services • There are multiple competing activities from Microsoft and IBM to Apache, IU and Anabas (for example) developing system and application services • Unlike for RTI and CORBA, services from different vendors should interoperate Container System Processing H 1 H 2 H 3 H 4 Body F 1 F 2 F 3 F 4 Container Handlers Service August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 2

What do Grids Add? • Grids use all of the Web Services • They address management and deployment of large distributed systems of services • They address security and management issues of virtual organizations crossing multiple administrative domains • GGF is developing specific services of relevance including job management, many aspects of data and scheduling • GGF has a good process for developing new higher level specifications – For example GGF 15 will address Cross enterprise security, Campus Grids, Enterprise Grids, High Performance messaging, GIS Grids, Portals and continue OGSA work integrating data, compute and management August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 3

The Grid and Web Service Institutional Hierarchy 4: Application or Community of Interest Specific Services such as “Run BLAST” or “Look at Houses for sale” 3: Generally Useful Services and Features Such as “Access a Database” or “Submit a Job” or “Manage Cluster” or “Support a Portal” or “Collaborative Visualization” OGSA and other GGF/W 3 C/ ……… WS-* from Handlers like WS-RM, Security, Programming Models like BPEL OASIS/W 3 C/ Industry 2: System Services and Features or Registries like UDDI 1: Container and Run Time (Hosting) Environment Apache Axis. NET etc. August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 4

The Grid and Web Service Functional Hierarchy G: User Interface F: Portal: Aggregation, Profiles E: Manipulating and Linking Services D: Brokering Monitoring and Managing Resources and Services C: Electronic Proxy Services for Resources B: Resources A: Pervasive System Services: Security, Collaboration, Messaging, Metadata August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 5

Composing Functionality and Resources in the Grid of Grids Methods CPUs Services Clusters MPPs Databases Sensor Federated Databases Functional Grids Compute Resource Grids Overlay and Compose Grids of Grids Data Resource Grids Sensor Nets August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 6

Critical Infrastructure (CI) Grids built in composite fashion and linked to an NCOW (Gi. G) Grid Flood CIGrid … Electricity CIGrid … Flood Services and Filters Collaboration Grid Sensor Grid Registry Security Portals GIS Grid Data Access/Storage Notification Workflow Physical Network NCOW Grid Do. D Services and Filters Visualization Grid Compute Grid Metadata Messaging August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 7

Port Mediation and Transformation in a Grid of Grids and Simple Services Port Internal Port Interfaces August 25 2005 Briefing Subgrid or service Messaging Port Subgrid or service Port Internal Port Interfaces Port External facing Interfaces Mediation and Transformation Services Subgrid or gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu service 8

ANABAS Do. D Services as Part of Grids August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 9

The Global Information Grid Core Enterprise Services Service Functionality CES 1: Enterprise Services Management (ESM) including life-cycle management CES 2: Information Assurance (IA)/Security Supports confidentiality, integrity and availability. Implies reliability and autonomic features CES 3: Messaging Synchronous or asynchronous cases CES 4: Discovery Searching data and services CES 5: Mediation Includes translation, aggregation, integration, correlation, fusion, brokering publication, and other transformations for services and data. Possibly agents CES 6: Collaboration Provision and control of sharing with emphasis on synchronous real-time services CES 7: User Assistance Includes automated and manual methods of optimizing the user Gi. G experience (user agent) CES 8: Storage Retention, organization and disposition of all forms of data CES 9: Application Provisioning, operations and maintenance of applications. August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 10

The Ten areas covered by the core WS-* Specifications WS-* Specification Area Examples 1: Core Service Model XML, WSDL, SOAP 2: Service Internet WS-Addressing, WS-Message. Delivery; Reliable Messaging WSRM; Efficient Messaging MOTM 3: Notification WS-Notification, WS-Eventing (Publish-Subscribe) 4: Workflow and Transactions BPEL, WS-Choreography, WS-Coordination 5: Security WS-Security, WS-Trust, WS-Federation, SAML, WS-Secure. Conversation 6: Service Discovery UDDI, WS-Discovery 7: System Metadata and State WSRF, WS-Metadata. Exchange, WS-Context 8: Management WSDM, WS-Management, WS-Transfer 9: Policy and Agreements WS-Policy, WS-Agreement 10: Portals and User Interfaces WSRP (Remote Portlets) August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 11

Activities in Global Grid Forum Working Groups GGF Area Standards Activities 1: Architecture High Level Resource/Service Naming (level 2 of fig. 1), Integrated Grid Architecture 2: Applications Software Interfaces to Grid, Grid Remote Procedure Call, Checkpointing and Recovery, Interoperability to Job Submittal services, Information Retrieval, 3: Compute Job Submission, Basic Execution Services, Service Level Agreements for Resource use and reservation, Distributed Scheduling 4: Database and File Grid access, Grid FTP, Storage Management, Data replication, Binary data specification and interface, High-level publish/subscribe, Transaction management 5: Infrastructure Network measurements, Role of IPv 6 and high performance networking, Data transport 6: Management Resource/Service configuration, deployment and lifetime, Usage records and access, Grid economy model 7: Security Authorization, P 2 P and Firewall Issues, Trusted Computing August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 12

Core Services in Action I NCOW Service or Feature WS-* Service area GGF and Others Demo Status A: General Principles Use Service Oriented Architecture Core Service Model (#1) Build Grids Services on Web Grid of Grids Composition YES DTA 7 B: NCOW Core Services (to be continued) CES 1: Enterprise Services Management WS-* #8 Management GGF #6: Management CIM Phase II (DTA 9) CES 2: Information Assurance(IA)/Security WS-* #5 WS-Security GGF #7, Grid-Shib, Permis Liberty Alliance etc. Phase II (Not in DTA) CES 3: Messaging WS-* #2, #3 JMS, MQSeries, Streaming /Sensor Technologies SBIR (DTA 1) CES 4: Discovery WS-* #6 CES 5: Mediation WS-* #4 workflow Treatment of systems. Transformations Legacy Data SBIR (DTA 6 DTA 7) CES 6: Collaboration VO GGF VO. XGSP, Shared Web Service ports SBIR (DTA 2 DTA 3) CES 7: User assistance WS- * #10 Portlets, NCOW Interfaces SBIR (DTA 8) JSR 168 SAB (DTA 10) Capability August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 13

Core Services in Action II NCOW Service or Feature WS-* Service area GGF and Others Demo Status B: NCOW Core Services Continued CES 8: Storage (not real-time streams) GGF #4 Data NCOW Data Strategy Phase II (Not in DTA) CES 9: Application GGF #2 ; Best Practice in building Grid/Web services SAB (DTA 6 DTA 7) Environmental Services ECS Control WS-*, #9 Resource Infrastructure Phase II (Not in DTA) GGF #5; Ad-hoc networks important NO C: Key NCOW Capabilities not directly in CES Meta-data WS-* #7 Resource/Service Matching/Scheduling Distributed Scheduling and SLA’s (GGF # 3) SAB (DTA 8) GGF scheduling work extended to networks Phase II with scheduled messaging (DTA 1) Sensors (real-time data) OGC Sensor standards SBIR (DTA 6) GIS OGC GIS standards SBIR (DTA 4 DTA 5) August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 14

Analysis of Grid Technology for Do. D • We developed four significant documents • Do. D Grid Opportunities for the Gi. G and NCOW – Grids for the Gi. G and Real Time Simulations (Proceedings of Ninth IEEE International Symposium DS-RT 2005 on Distributed Simulation and Real Time Applications' Montreal October 10 -12 2005) is subset • Appendix on Grid and Web Services covering all basic Grid and Web services • Grid Technology Overview and Status reviews Grids • Grid Application Areas within Do. D is a general study of applications of Grids in Do. D • Complete list of References for Reports on Grids for Gi. G and NCOW contains all references in above reports • Implementing some Grid Application Areas within NCOW 1. 1 of Do. D is not started but is intended to write up specific applications such as demos August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 15

Major Conclusions I • One can map “broadly” NCOW and Gi. G core capabilities into Web Service (WS-*) and Grid (GGF) architecture and core services – Analysis of Grids in NCOW document inaccurate (confuse Grids and Globus and only consider early activities) • Important “mismatches” on both NCOW and Grid sides • Grid/WS-* do not have messaging and collaboration • NCOW does not have system metadata and resource/service scheduling and matching • Higher level services of importance include GIS (Geographical Information Systems), Sensors and datamining • Criticisms of Web services in a recent paper by Birman seem to be addressed by Grids or reflect immaturity of initial technology implementations August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 16

Major Conclusions II • NCOW does not seem to have any analysis of how to build their systems on WS-*/Grid technologies in a layered fashion; they do have a layered service architecture so this can be done – In particular do not clearly endorse critical header/body structure of SOAP messages • Grid of Grids allows modular architectures and natural treatment of legacy systems August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 17

Areas in Birman Paper • • Information Architecture and Service Description Document-Centric Nature Time-critical Events Life Cycle Support for Services Reliable Messages Security Scalability August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 18

Grids and HLA/RTI I • HLA through IEEE 1516 has specified the interfaces for its key services that are supported by RTI (Run Time Infrastructure) • HLA does not specify each message semantics or core system services – RTI implementations are NOT interoperable although each one should support any HLA federation – RTI implementations become a full distributed system environment as need metadata, reliable messaging etc. with simulation support only a small part • Grids can be used in HLA with – Dynamic assignment of compute resources to support federates – Building web service interfaces to federates (XMSF) – Infrastructure to build a new generation of RTI that will use Web system services and just add simulation support August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 19

Grids and HLA/RTI II • HLA specifies – Declaration management – achieved through use of publish/subscribe Grid Messaging (Narada. Brokering) – Data Distribution management – corresponds to geometry sensitive publish and subscribe model (add to Narada. Brokering) – Time management – corresponds to simulation framework (use best event driven and time stepped models – as infrastructure generic, one can support broad range of simulations including classic parallel computing and agent-based simulations) – Object management - Very specific to HLA and should be built as per IEEE 1516 – Ownership management - could use Grid virtualization and use metadata catalogs to handle properties – might be generalizable – Federation management - Could generalize to support of general simulation models (federates and federations are a general concept) August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 20

Demo Technology Areas DTA 1) Narada. Brokering: Grid Messaging 2) Anabas Collaborative Applications 3) Global. MMCS/XGSP Collaborative Stream Management Framework and A/V Conferencing 4) Geographical Information Systems (GIS) Web Feature Services (WFS) 5) GIS Web Map Services (WMS) and Clients 6) GIS Sensor Enablement Services 7) Wrapping of non Grid Applications as Grids of grids or Grids of Services 8) Information and meta-data Services 9) HPSearch Workflow and Management 10) Portals and Portlets August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 21

IN 1 Data Mining and GIS Grid Databases with NASA, USGS features SERVOGrid Faults WFS 1 UDDI Data Mining Grid WFS 3 WFS 2 NASA WMS handling Client requests SOAP WMS Client HTTP August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 22

California fault data from Quake Tables fault database via Web Feature Service. August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 23 Standard Open Geospatial Consortium WMS Clients

Standard Open Geospatial Consortium WMS Clients Get Feature Info allows users to get map information. This can also be used to read feature info off the map when creating input data for applications August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 24

IN 1 Data Mining Grid Databases with NASA, USGS features SERVOGrid Faults UDDI WFS 4 SOAP Pipeline Filter PI Data Mining HPSearch Workflow Narada Brokering System Services Filter WS-Context WFS 3 GIS Grid August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 25

PI demo combines WFS, WMS, and HPSearch for service orchestration. Tool bar items allows you to adjust maps Users set up problems by adding filtered seismic archives from 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu WFS as. August map 25 layers. 26

Hot spots calculations-areas of increased earthquake probability in the forecast time-calculations are re-plotted on the map as features. August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 27

Typical use of Grid Messaging in NASA Sensor Grid Eventing Datamining Grid GIS Grid August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 28

August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu

Typical use of Grid Messaging Sensor Grid Post before Processing Filter or Datamining Post after Processing Narada Brokering Notify WFS Subscribe Database Archivess WS-Context Stores dynamic data HPSearch Manages August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 30

IN 2 Google Map Client Archived Real Time Databases with SERVOGrid Faults Sensor Grid WFS 1 WFS 2 Helper Services SOAP Google Central HTTP Google Map Client UDDI August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 31

IN 2: Real Time GPS and Google Maps Subscribe to live GPS station. Position data from SOPAC is combined with Google map clients. Select and zoom to GPS station location, click icons for more information. August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 32

IN 2: Integrating Archived Web Feature Services and Google Maps Google maps can be integrated with Web Feature Service Archives to filter and browse seismic records. August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 33

IN 2: Integrating Archived Web Feature Services and Google Maps Google maps can be integrated with Web Feature Service Archives to browse earthquake fault records. Faults are typically stored by segment number, so map interfaces are convenient for both verifying continuity 34 and up input files for computing problems. August 25 2005 setting Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu

IN 2: Same Screens with Hybrid Maps August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 35

IN 3 Google Maps as a WMS Web service Databases with NASA, USGS features SERVOGrid Faults WFS 1 UDDI WFS 2 Google Central …. . Google Maps WMS (Gateway) WMS handling Client requests SOAP Other Services using results WMS Client HTTP August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 36

Google Maps as Service accessed from our WMS Client August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 37

IN 4 Server-side Streaming from a WMS Databases with NASA, USGS features SERVOGrid Faults WFS 1 WFS 2 Data Mining Grid WFS 3 NASA WMS SOAP UDDI XGSP Media Service WMS handling Client requests Collaborative Streaming Grid Narada Brokering August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 38

IN 4 Shared Display Streaming from a WMS Databases with NASA, USGS features SERVOGrid Faults WFS 1 WFS 2 Data Mining Grid WFS 3 NASA WMS SOAP UDDI WMS handling Client requests WMS Client Collaborative Narada Streaming Brokering Grid August 25 2005 Briefing GMC = Global. MMCS Client XGSP Media Service gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 39

Web Service Collaboration Shared Output port with replicated recipients Web Service Narada Brokering Shared Input Port with replicated services WS 1 Narada Brokering WS 2 WS 3 August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 40

Pipelined Web Service Collaboration • In a workflow, one can invoke collaborative streams on any flow and this splitting is between output port of one and input of next Web Service in chain WS-A WS-B Narada Brokering WS 1 WS 4 WS 2 WS 5 WS 3 WS 6 Shared Output Port Shared Input Port August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 41

Collaboration Grid WS-Context Narada Broker HPSearch XGSP Media Service Audio Mixer Video Mixer UDDI Narada Broker WS-Security Gateway Narada Broker Gateway ay w e t Ga Shared. WS y Gatewa Transcoder Thumbnail Replay Record Annotate Shared. Display White. Board August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 42

August 25 2005 Briefing gcf@iu. edu Annotation of GIS Maps from WMS converted into H 261 Video Stream 43 alexho@anabas. com mpierce@cs. indiana. edu

GIS TV Chat Video Mixer Webcam New Global. MMCS Client August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 44

Collaborative Map Pages • Shared Input Port Google Central Anabas Endpoint XML Sharedlet interface similar to WSDL Anabas Endpoint Narada Brokering Anabas Endpoint Input to Google web page August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 45

Impromptu Google Interface August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 46

Impromptu Google Map on Whiteboard August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 47

Clients and Services • Thick clients can be Grid (WSDL) endpoints and act as services Grid Service SOAP Client • Can break thick client up as MVC with Model becoming a service – Message-based MVC Portlet Model as Service Grid Service SOAP View Portal HTTP August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 48

Collaborative “Legacy” Object • Shared Output Port with Anabas Endpoint providing Service wrapping of RMI Object Java Distributed Object RMI Anabas Endpoint XML Sharedlet interface similar to WSDL Anabas Endpoint Narada Brokering Anabas Endpoint Output from Java Object August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 49

Collaborative Annotation of Map Pages • Shared Input Port with replicated white board Google Central Anabas WB Service XML Sharedlet interface similar to WSDL Anabas Google Endpoint WB Service Narada Brokering Anabas WB Service White Board changes August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 50

Collaborative Search • Shared Input Port for annotation with replicated search display (shared output port for search service) Yahoo Search Anabas Annotatable Text Service Meta. Search Anabas Search Endpoint Annotatable Text Service Narada Brokering Google Search Anabas Annotatable Text Service Search Results and annotation August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 51

Impromptu Yahoo Search August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 52

August 25 2005 Briefing gcf@iu. edu Impromptu Google alexho@anabas. com mpierce@cs. indiana. edu Search 53

Impromptu Meta. Search August 25 2005 Briefing gcf@iu. edu alexho@anabas. com mpierce@cs. indiana. edu 54