Grids for Geo Sensors Geo Science and Geo

Grids for Geo. Sensors, Geo. Science and Geo. Scientists Earth. Scope CSIT Workshop March 25 2002 PTLIU Laboratory for Community Grids Geoffrey Fox Computer Science, Informatics, Physics Indiana University, Bloomington IN 47404 http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02 gcf@indiana. edu 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupag es/presentations/earthscopemar 02" email="gcf@indiana. edu" 1

Trends of Importance n Resources of increasing performance or functionality • Computers (ASCI, Earth Simulator to Tera. Grid), storage, sensors, networks, PDA’s n Applications of increasing sophistication • Size, multi-scales, multi-disciplines n New algorithms and mathematical techniques n Computer science • Compilers, Parallelism, Objects, Components n Grid and Internet Concepts and Technologies • Enabling new applications • XML, Web Services, Portals, Collaboration 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 2

Projected Top 500 Until Year 2009 n First, Tenth, 100 th, 500 th, SUM of all 500 Projected in Time Earth Simulator from Japan http: //geofem. tokyo. rist. or. jp/ 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 3

PACI 13. 6 TF Linux Tera. Grid 32 256 p HP X-Class 32 24 128 p HP V 2500 24 8 92 p IA-32 HPSS Calren NTON 574 p IA-32 Chiba City 8 24 Extreme Black Diamond 4 64 Nodes 1 TF 0. 25 TB Memory 25 TB disk 32 Nodes 0. 5 TF 0. 4 TB Memory 86 TB disk 32 32 5 32 128 p Origin 32 HR Display & VR Facilities 5 HPSS OC-12 Chicago & LA DTF Core Switch/Routers Cisco 65 xx Catalyst Switch (256 Gb/s Crossbar) OC-48 OC-12 ATM v. BNS Abilene Calren ESnet Argonne Caltech Juniper M 40 OC-12 2 OC-12 Gb. E Juniper M 160 OC-3 SDSC NCSA 256 Nodes 4. 1 TF, 2 TB Memory 225 TB disk 500 Nodes 8 TF, 4 TB Memory 240 TB disk ESnet HSCC MREN/Abilene Starlight Juniper M 40 OC-12 2 v. BNS Abilene MREN OC-12 OC-3 8 4 2 Sun Starcat 4 1176 p IBM SP Blue Horizon 4 Sun E 10 K Uni. Tree 8 HPSS = 32 x 1 Gb. E 1024 p IA-32 320 p IA-64 16 Myrinet Clos Spine = 64 x Myrinet = 32 x Myrinet 14 Myrinet Clos Spine = 32 x Fibre. Channel 1500 p Origin = 8 x Fibre. Channel 10 Gb. E 12/3/2020 32 quad-processor Mc. Kinley Servers (128 p @ 4 GF, 8 GB memory/server) 32 quad-processor Mc. Kinley Servers (128 p @ 4 GF, 12 GB memory/server) Fibre Channel Switch 16 quad-processor Mc. Kinley Servers (64 p @ 4 GF, 8 GB memory/server) Cisco 6509 Catalyst Switch/Router IA-32 nodes uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 4

Small Devices Increasing in Importance n n n There is growing interest in wireless portable displays in the confluence of cell phone and personal digital assistant markets By 2005, 60 million internet ready cell phones sold each year 65% of all Broadband Internet accesses via non desktop appliances 12/3/2020 CM 5 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 5

The HPCC Track n n The 1990 HPCC 10 year initiative was largely aimed at enabling large scale simulations for a broad range of computational science and engineering problems It was in many ways a success and we have methods and machines that can (begin to) tackle most 3 D simulations • ASCI simulations particularly impressive • Do. E still putting substantial resources into basic software and algorithms from adaptive meshes to PDE solver libraries Machines are still increasing in performance exponentially and should achieve petaflops in next 7 -10 years Earth. Scope community needs to harness these capabilities • Japan’s Earth Simulator activity major effort with large hardware and software (GEOFEM) efforts 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 6

Some HPCC Difficulties n n An Intellectual failure: we never produced a better programming model than message passing • HPCC coding is hard work • Successes of ASCI software like “Grid FTP” – not parallelizing compilers An institutional problem: we do not have a way to produce complex sustainable software for a niche (1%) market like HPCC. • POOMA support just disappeared one day (foundation of first proposal GEM wrote) • One must adopt commodity standards and produce “small” sustainable modules. • Note distributed memory becoming dominant again with bizarre clustered SMP architecture – not clear that “wise” to exploit advantages of shared memory architectures 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 7

HPCC Advice to Earth. Scope n n n n KISS: Keep it Simple and Sustainable Use MPI and open. MP if needed for performance on shared memory nodes Adaptive Meshes Are well understood Load Balancing to get high performance parallel simulations PDE Solvers including fast multipoles Use broad community expertise Particle dynamics Other areas such as datamining, visualization and data assimilation quite advanced but still significant research 12/3/2020 } uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 8

Use of Object Technologies I n n n The claimed commercial success in using Object and component technology has not yet been a clear success in HPCC • Object technologies do not naturally support either high performance or parallelism • C++ can be high performance but Java (as a language) is not uniformly so (it is improving) • Web Services could change this Fortran (including Fortran 90) will continue to decline in importance and interest – the community should prefer not to use it • It’s use will not attract the best students Not essential to write modules in object oriented language • It is essential to package modules in object framework 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 9

Use of Object Technologies II n n There is emerging HPCC component architecture allowing production of more modern libraries (integration Infrastructure) • Do. E has very large CCA – Common Component Architecture – effort • Package software (“system and applications”) as distributed objects – not as traditional libraries CORBA Java and Web Services are not naturally high performance as component models • High performance often not essential for coarse grain objects • Web Services support multiple implementations allowing performance functionality trade-off 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 10

Application Structure n n n Earth Science applications are typically multi-scale and multidisciplinary • i. e. a given simulation is made of multiple components with either different time/length scales and/or multiple authors from possibly multiple fields I am not aware of a systematic “Computational renormalization group” – a methodology that links different scales together However composition of modules is an area where (component) technology of growing sophistication is becoming available • Needed commercially to integrate corporate functions • Easiest for large coarse grain components • Integration of data and simulation is one example of fine-scale composition which is “understood” 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 11

Object Size & Distributed/Parallel Simulations n n n All interesting systems consist of linked entities • Particles, grid points, people or groups thereof Linkage translates into message passing • Cars on a freeway • Phone calls • Forces between particles Amount of communication tends to be proportional to surface area of entity whereas simulation time proportional to volume So communication/computation is surface/volume and decreases in importance as entity size increases In parallel computing, communication synchronized; in distributed computing “self contained objects” (whole programs) which can be scheduled asynchronously 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 12

Some Problem Classes n Classic HPCC: synchronized objects with regular time structure (communication overhead decreases as problem size increases) • Includes PDE and interacting particle based applications • Give scaling parallelism on large MPP’s n Internet Technology and Commercial Application Integration: Large objects with modest communications and without difficult time synchronization • Compose as independent (pipelined) services • Includes some approaches to multi-disciplinary simulation linkage n Hardest: smallish objects with irregular time synchronization • Interesting Los Alamos SDS technology for this 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 13

What is a Grid or Web Service? n n There are generic Grid system services: security, collaboration, persistent storage, universal access • OGSA (Open Grid Service Architecture) is implementing these as extended Web Services An Application Web Service is a capability used either by another service or by a user • It has input and output ports – data is from sensors or other services Consider Satellite-based Sensor Operations as a Web Service • Satellite management (with a web front end) • Each tracking station is a service • Image Processing is a pipeline of filters – which can be grouped into different services • Data storage is an important system service • Big services built hierarchically from “basic” services Portals are the user (web browser) interfaces to Web services 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 14

Sensor Web Service Out Web Service ports Universal sensor access for people/computers 12/3/2020 Distributed Sensor Web Service In Web Service ports Different format Sensor Data uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 15

Application Web Services Prog 1 n Filter 1 Prog 2 Filter 2 integrates. Filter 3 Note Service model sensors, sensor analysis, simulations and people WS WS WS n An Application Web Service is a capability used either by another service or by a user Build as multiple Filter Web Services • It has input and output ports – data is from users, interdisciplinary sensors or other services Programs • Big services built hierarchically from “basic” services Sensor Data as a Web service (WS) Simulation WS Data Analysis WS Sensor Management WS 12/3/2020 Visualization WS SLE (space Link Extension) as a WS uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 16

The Application Service Model n n n As bandwidth of communication (between) services increases one can support smaller services A service “is a component” and is a replacement for a library in case where performance allows Services (components) are a sustainable model of software development – each service has documented capability with standards compliant interfaces • XML defines interfaces at several levels • WSDL at Service interface level and XSIL or equivalent for scientific data format A service can be written as Perl, Python, Java Servlet, Enterprise Javabean, CORBA (C++ or Fortran) Object … Communication protocol can be RMI (Java), IIOP (CORBA) or SOAP (HTTP, XML) …… 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 17

Services support Communities n n Grid Communities (Earth Science, SCEC, Do. D, Earth Science, High School Classes) are groups of communicating individuals sharing resources implemented as Web Services Access Grid from Argonne/NCSA is high-end Audio/Video conferencing technology Peer to Peer networking describes a set of technologies supporting community building with an emphasis on less structured groups than classic “users of a supercomputer” Peer to peer Grids combine the technologies and support “small worlds” – optimized networks with short links between each community member 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 18

e-Science is just a pile of XML n Each leaf is a piece of XML either defining a nugget of information and/or containing links to other XML or “raw resources” Database 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 19

Biased History of Computing n n In almost the beginning, there was Fortran and formats (6 I 5, 5 F 10. 4) for data ……………. . 1993 -1997: HTML came along for Web Pages 1998 -…: XML was developed to define information in documents while HTML defining rendering • But soon it became used for specifying all data and their format n n 2001: Web Services allowed XML to specify methods (subroutines) as well as data Java, C++, Python, Perl, . . Fortran are now “just” the insides of XML specified programs 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 20

XML (RSS) Specification of Information Nuggets n n n n n <item rdf: about="http: //xml. com/pub/2000/08/09/xslt. html"> <title> Processing Inclusions with XSLT </title> <link> http: //xml. com/pub/2000/08/09/xslt. html </link> <description> Processing document inclusions with general XML tools can be problematic. This article proposes a way of preserving inclusion information through SAX-based processing. </description> </item> <item rdf: about="http: //xml. com/pub/2000/08/09/rdfdb/index. html"> <title> Putting RDF to Work </title> <link> http: //xml. com/pub/2000/08/09/rdfdb/index. html </link> <description> Tool and API support for the Resource Description Framework is slowly coming of age. Edd Dumbill takes a look at RDFDB, one of the most exciting new RDF toolkits. </description> </item> Example of XML meta-data in the “pile” </rdf: RDF> pointing to other (outside) resources 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 21

What is a Web Service I n n A web service is a computer program running on either the local or remote machine with a set of well defined interfaces (ports) specified in XML (WSDL) In principle, computer program can be in any language (Fortran. . Java. . Perl. . Python) and the interfaces can be implemented in any way what so ever • Interfaces can be method calls, Java RMI Messages, CGI Web invocations, totally compiled away (inlining) but n n n The simplest implementations involve XML messages (SOAP) and programs written in net friendly languages like Java and Python Web Services separate the meaning of a port (message) interface from its implementation Enhances/Enables Re-usable component model of ANY electronic resource 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 22

Raw Resources Raw Data (Virtual) XML Data Interface Web Service (WS) WS WS WS etc. XML WS to WS Interfaces WS WS WS (Virtual) XML Knowledge (User) Interface Render to XML Display Format Clients 12/3/2020 (Virtual) XML Rendering Interface uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 23

Classic Grid Architecture Resources Database Composition Content Access Netsolve Security Collaboration Middle Tier Brokers Service Providers Computing Middle Tier becomes Web Services 12/3/2020 Clients Users and Devices uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 24

What is a Web Service II n n n Web Services have important implication that ALL interfaces are XML messages based. In contrast Most Windows programs have interfaces defined as interrupts due to user inputs Most software have interfaces defined as methods which might be implemented as a message but this is often NOT explicit WSDL interfaces Security WSDL interfaces 12/3/2020 Payment Credit Card Catalog Warehouse shipping uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 25

What is a Web Service III n n n “Everything electronic” is a resource • Computers; Programs; People • Data (from sensors to this presentation to email to databases) “Everything electronic” is a distributed object All resources have interfaces which are defined in XML for both properties (data-structure) and methods (service, function, subroutine) (Resources are Services) • We can assume that a data-structure property has getproperty() and setproperty(value) methods to act as interface All resources are linked by messages with structure, which must be specifiable in XML All resources have a URI such as unique: //a/b/c ……. 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 26

WSDL Abstractions n n WSDL abstracts a program as an entity that does something given one or more inputs with its results defined by streams on one or more outputs. Functions are defined by method name and parameters methodname(parm 1, parm 2, … parm. N) • Where parameters are “Input” “Output” or both n In WSDL, we will have a Web Service which like a (Java or CORBA Program) can be thought of as a (distributed) object with many methods • Instead of a function call, the “calling routine” sends an XML message to the Web Service specifying methodname and values of the parameters • Note name of function is just another parameter 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 27

Details of WSDL Protocol Stack n n n n UDDI finds where programs are • remote( (distributed) programs are just Web Services WSFL links programs together (under revision? ) WSDL defines interface (methods, parameters, data formats) SOAP defines structure of message including serialization of information HTTP is negotiation/transport protocol TCP/IP is layers 3 -4 of OSI Physical Network is layer 1 of OSI 12/3/2020 UDDI or WSIL WSFL WSDL SOAP or RMI HTTP or SMTP or IIOP or RMTP TCP/IP Physical Network uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 28

Examples of Web Services I n n OGSA (Open Grid Service Architecture) • Integrate Web Service and Grid Concepts and allows Globus to be implemented as Web Services Audio-Video Conferencing as a Web Service • Integrates H 323, SIP, JXTA (etc. ) protocols by mapping to single XML Interface • Provides VRVS reflector model from Messaging Web Service Messaging or Event Web Service provides intelligent routing and buffering of messages Computing as a Web service • Job submittal, status, composition, data services, visualization • Performance WS allows access to distributed monitoring data, analysis, models, and final benchmarks with interoperable XML interfaces 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 29

Examples of Web Services II n Education as a Web Service • One of easiest to do as object standards well defined (IMS) and little performance issues • Grading, Homework submission, registration, assessment etc. n Universal Access and Web Services • As Web Services allow multiple implementation of a particular interface, one can adjust to needs of particular clients (PDA v. versus, impaired sight etc. ) • Can build custom implementations of certain web services for particular communities but re-use others n Collaborative Web Services • As interfaces all message based, much easier to share Web Services than other applications (Power. Point interface is NOT message based and harder to share than server app) 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 30

Education as a Web Service n n n n Can link to Science as a Web Service and substitute educational modules “Learning Object” XML standards already exist from IMS/ADL http: //www. adlnet. org – need to update architecture Web Services for virtual university include: Registration Performance (grading) Authoring of Curriculum Online laboratories for real and virtual instruments Homework submission Quizzes of various types (multiple choice, random parameters) Assessment data access and analysis Synchronous Delivery of Curricula Scheduling of courses and mentoring sessions Asynchronous access, data-mining and knowledge discovery Learning Plan agents to guide students and teachers 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 31

Distributed Information Actually the XML is distributed all around in a dynamic Grid 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 32

Structured (XML) Information earthscope: //root/one/two/bottom Note XML specifies root both internal and external nodes of tree one two bottom 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 33

Matching Information/Service Providers and Consumers I n n n Classic Centralized Approach Those with services publish information as to location – this is percolated up and down the tree of brokers At simplest, publish location; better publish location and meta-data allowing easier discovery of value Those wanting service, look it up using either • Some search of information registered with brokers • A search using a system like Google • Because they were told some key Like using an encyclopedia; very reliable and fast for well established information 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 34

Unstructured and Structured XML Hoosier National Forest showing structured trees and a Gallimaufry of unstructured leaves (fall 2001) earthscope: //root/one/two/mess root one two mess “mess” can be multiple levels of tree 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 35

Database JXTA Database Web Service Interfaces Event/ Message Brokers Integrate P 2 P and Grid/WS Event/ Message Brokers Peer to Peer Grid Web Service Interfaces JXTA 12/3/2020 36 Peer to Peer Grid uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu"

Matching Information/Service Providers and Consumers II n n n Peer-to-peer Approach (or how to search the “mess”) Those with services publish XML advertisements to their friends; their friends may forward it to other friends Those wanting a service, publish an XML request to a chosen set of friends Friends use their personal idiosyncratic approach to matching requests with advertisements and to choosing who else should be asked Analogous to way communities exchange information as in a meeting like this Uncertain reliability but scales well (communities intra-exchange information independently and supports rapidly varying information (Web Services) 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 37

XML Skin Message Or Event Soft Based ware Inter Connection Resource XML Skin Resource Data base e-Science is XML Specified Resources connected by XML specified messages 12/3/2020 Implementation of resource and connection may or may not be XML uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 38

Technology Trends and Principles n n n All performance and capability measures of infrastructure continue to improve Gilder’s law says that network bandwidth increases 3 times faster than CPU Performance (Moore’s Law) The Telecosm eclipses the Microcosm (but don’t look at Wall Street) …. George Gilder Telecosm : How Infinite Bandwidth Will Revolutionize Our World (September 2000, Free Press; ISBN: 0684809303, #146(3883) in Amazon Sales Jan 15 2001(July 29 2001)) 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 39

Grid/P 2 P Use of Internet I Rival Estimate Mainly Digital Video Cohen’s ROBERT B. COHEN, PH. D. COHEN COMMUNICATIONS GROUP bcohen@bway. net 212 -986 -7720 12/3/2020 email="gcf@indiana. edu" Global Grid uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" Forum Toronto Feb 18 2002 40

Grid/P 2 P Use of Internet II S 2 S Server to Server Digital Video “on demand” 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 41

Meta-Data and Web Services n Enriching resources with meta-data is critical idea • Enables one to identify and link resources around the globe • Allows one to find out “meaning” of a Web service not just syntax of interface n n n Semantic Grid implies linkage of Grid/Web services enabled by meta-data leading to “digital brilliance” phase transition We can experiment with Semantic Web techniques for specifying meta-data RDF DAML OIL These encompass both straightforward enriched data as well as Artificial Intelligence assertions 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 42

Semantic Grid & Digital Brilliance I n n The (XML) advertisement-request matching provides a publish-subscribe linkage between resources – these are people, computers and raw/processed data The richer the meta-data, the more precise the linkage • This is spirit of Semantic Web – RDF/DAML/OIL metadata enables meaningful linkage In a physics analogy, resources can be thought of as spins and the meta-data induced linkage as interactions Phase transitions will occur when “enough” resources are linked – one will get associated spins to align in the direction of new knowledge • Term this digital brilliance 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 43

Semantic Grid & Digital Brilliance II n n This suggests ways of quantifying value of metadata induced linkages and ways of identifying where one “should” add more resource specifications Note that related resources are not necessarily directly connected but rather messages are forwarded through friends Study of Peer to Peer networks teach us that we can build “small worlds” where distance between resources is logarithmic in number of nodes This physics based picture provides an interesting underlying formalism to give a theory of e-Science …. • All you need to do is to build a lot of XML Meta-data specification wizards 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 44

Semantic Grid & Digital Brilliance III n n Earth. Scope Collaboratory consists of a set of connected “spins” (being a physicist; resources if I was W 3 C) Resources are anything with a digital signature • Raw data, Analysers, Simulators, Simulations, Processed Information, Extracted Knowledge, Scientists …. n n n The linkage of Earthquake Fault Simulator Web Service to the Greens Function Solver Web Service is as program to subroutine; must have agreement on both syntax and Semantics The linkage of Granular Physics model to (my) remark that Los Alamos has interesting new simulation technology is less precise So linkages with very precise ontologies and those which are more qualitative are both part of Semantic Grid 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 45

Directory mode for Google 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 46

Web (Unstructured) mode for Google Quite Similar to Directory mode 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 47

Portals and Web Services n n Web Services allow us to build a component model (see CCA) for resources. Each resource naturally has a user interface (which might be customized for user) Web Service <--> Portlet Natural to use a component model for portal building displayed web page from collection of portlets • So can customize each portlet and customize which portlets you want n Apache Jetspeed seems good open source technology supporting this model • JSP model is better than say a client-side Java integration in that also message based so this is “Portal as a Web Service” 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 48

Jetspeed Computing Portal: Choose Portlets 4 available portlets linking to Web Services I choose two 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 49

Choose Portlet Layout Choose 1 -column Layout Original 2 -column Layout 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 50

The File Browser Portlet 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 51

Two Computing Portlets 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 52

Online Knowledge Center for Do. D HPCMO n n n Web Services provide a component model for the middleware (see large “common component architecture” effort in Dept. of Energy) Should match each WSDL component with a corresponding user interface component Thus one “must use” a component model for the portal with again an XML specification (portal. ML) of portal component 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 53

Earth. Scope CSIT Strategy n Make a list of resources with a hierarchical arrangement • People, Places, Results (Publications, meeting archives, Simulation Output), Activities, Sensors (Instruments), Data (raw and processed), Earth features, Computers, Software n Decide on component (Web Service) model and URI labelling (earthscope: //devices/satellites/year/label …) • Respect performance requirements • Design so modules can be re-used, re-arranged and replaced for outreach (education) n Study related CSIT architectures of other fields • • 12/3/2020 Grid Forum, PACI, ASCI for computing issues W 3 C Web Consortium for basic IT infrastructure open. GIS XMML for related fields IMS for Education uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 54

Earth. Scope HPCC Strategy n n Decide what services are well enough understood and useful enough to be encapsulated as application Web Services • Parallel FEM Solvers • Visualization • Parallel Particle Dynamics • Access to Sensor Data • Image Processing Make services as small as possible – smaller is simpler and more sustainable but with higher communication needs • Compose large services from smaller ones Design Portals and portal components that allow one to manipulate services – set parameters, compose, invoke Implement chosen System Web Services (job submit, performance, queue) on central machines and local clusters • Make certain infrastructure supports compute, data, middleware needs • Set necessary hardware/software meta-data 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 55

n n n Earth. Scope IT Strategy Design an internal EIF (Earth. Scope Internal Framework) defining architecture and interface standards of internal Web Services and data structures Design EEF (Earth. Scope External Framework) which maps external raw data into sensor web services Choose some appropriate (mix of) middleware frameworks • . net, IBM, BEA, Sun, Oracle n Look at special requirements for key system services • • n Hardware/Data systems (new and legacy issues) Security Collaboration including Audio/Video conferencing Peer-to-peer networking Develop necessary meta-data wizards 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 56

Earth. Scope Peer to Peer Grid Community APAN Network linking 12/3/2020 email="gcf@indiana. edu" Accessuri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" Grids 57

Gateway and Web Services n We will use the Gateway Computing Portal as an example (http: //www. gatewayportal. org) • It is largely built using CORBA with a Java Server Pages front end • http: //community. ucs. indiana. edu: 8004/GCWS/Batch. Script. Gen/Main. jsp n Several capabilities have been interfaced using WSDL • Job Submission (11 Methods including execute local and remote command, copy files etc. as well as Submit Job) • Manage Web. Flow Session (67 Methods) • Generate Batch Script (just 1 method but two implementations developed – one at SDSC and one at Indiana – with UDDI to manage) • Each is one service – could have used finer grain services • Sample files are at http: //grids. ucs. indiana. edu/ptliupages/presentations/ggf 4 feb 02 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 58

Gateway Architecture As needed Do. D approved Kerberos/Secure. ID security, does not usually link to standard Grid technology (Globus) n Pre WSDL Post WSDL backend Globus or submit to local queue system CORBA Corba middle tier SECIOP with Java modules JSP SSL 12/3/2020 Add Castor Java to XML Will replace historical CORBA By EJB WSDL SOAP wrapper JSP Servlet uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 59

WSDL from a Gateway View 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 60

12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 61

SOAP and Gateway Portal I n n n Having specified service in WSDL, the run-time is implemented in SOAP which is “just” an XML header (info needed by transport – empty here) and body Here is SOAP transported by HTTP message This is exec. Local. Command WSDL operation to run one particular command (ls) on current Web. Flow directory HTTP Header Argument of operation 12/3/2020 Specify ls as SOAP Envelope and body uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 62

HTTP Header SOAP Envelope and body SOAP and Gateway Portal II n 12/3/2020 And this is the result of ls sent back to client in SOAP over HTTP uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 63

WSDL Message Example <message name="submit. Request"> <part name="xmljob" type="xsd: string"/> </message> <message name="submit. Response"> <part name="response" type="xsd: string"/> </message> For the batch script service, we pass the XML description of the job as a string and get back the script as a string. In general, any XML primitive or complex types can be used in messages. We could improve our service by defining a Batch. Script complex type. 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 64

WSDL port. Types Example <port. Type name="Batch. Script. Service. Port. Type"> <operation name="batch. Gen"> <output message="tns: submit. Response" name="submit. Response"/> <input message="tns: submit. Request" name="submit. Request"/> </operation> </port. Type> A port. Type corresponds to a Java class, so if we compile this WSDL to make client stubs, we will generate a Batch. Script. Service. Binding. java class. 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 65

WSDL SOAP Binding Example <binding name="Batch. Binding" type="tns: Batch. Script. Service. Port. Type"> <soap: binding style="rpc" transport="http: //schemas. xmlsoap. org/soap/http"/> <operation name="batch. Gen"> <soap: operation soap. Action=""/> <input> <soap: body use="encoded“ namespace="urn: Batch. Script. Service" encoding. Style="http: //schemas. xmlsoap. org/soap/encoding/"/> </input> <output> <soap: body use="encoded" namespace="urn: Batch. Script. Service“ encoding. Style="http: //schemas. xmlsoap. org/soap/encoding/"/> </output> </operation> note binding’s “type” attribute points back </binding> to the port. Type tag by name 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 66

WSDL Ports and Services <service name="Batch. Script. Service"> <documentation>BS stands for Batch Script </documentation> <port binding="Batch. Binding” name="Batch. Port"> <soap: address location= "http: //yourserver/soap/servlet/rpcrouter/"/> </port> ports are concrete implementations of </service> and point back to a particular </definitions> port. Types binding (by name). They also point to the specific location of a server that implements the service. A service is a collection of one or more ports. 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 67

What is a Virtual XML Interface n n We can specify interfaces in XML but we are not required to implement in XML. Example 1: We aren’t likely to change syntax of mail Reply-to: Geoffrey Fox <gcf@indiana. edu> To: Geoffrey Fox <gcf@grids. ucs. indiana. edu> Subject: A Test for Tutorial A simple mail message Geoffrey Fox gcf@indiana. edu FAX 8128567972 Phones Cell 315 -254 -6387 Home 812323919 n n But we could specify and indeed store in XML with transport done using conventional SMTP. So conventional mail is easy to give a virtual XML specification for with name: value becoming <name>value</name> but this is not only way 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 68

Mail in XML n <mailasxml uri=“gxos: //mail/users/gcf/sent/2002/february/290” > <smtpheaders> <reply-to email=“gcf@indiana. edu” >Geoffrey Fox</reply-to> <to email=“gcf@grids. ucs. indiana. edu” >Geoffrey Fox</to> <subject>A Test for Tutorial</subject> </smtpheaders> <smtpbody> <message whitespace="collapse"> A simple mail message</message> <signature personuri=“ssn: //123/45/6789” whitespace=“preserve” > Geoffrey Fox gcf@indiana. edu FAX 8128567972 Phones Cell 315 -254 -6387 Home 812323919 </signature> </smtpbody> </mailasxml> n Such an interface could be used by “User Messaging as a Web service” which could perhaps integrate News Groups, e-mail, text chat, instant messenger, voicemail 12/3/2020 uri="http: //grids. ucs. indiana. edu/ptliupages/presentations/earthscopemar 02" email="gcf@indiana. edu" 69