Scripting based architecture for Management of Streams and

Scripting based architecture for Management of Streams and Services in Real-time Grid Applications Authors Harshawardhan Gadgil, Geoffrey Fox, Shrideep Pallickara, Marlon Pierce Community Grids Lab, Indiana University, Bloomington Robert Granat NASA, Jet Propulsion Lab, Pasadena Presented By Harshawardhan Gadgil hgadgil@cs. indiana. edu May 10, 2005 Community Grids Lab Presentation for CCGrid 2005

Talk Outline n Introduction to HPSearch Architecture n A quick view of Narada. Brokering middleware n System Goals – RDAHMM Example n Related May 10, 2005 and Future Work Community Grids Lab Presentation for CCGrid 2005 2

HPSearch n A Java. Script based scripting runtime serving as an administration console – Currently uses Rhino (http: //www. mozilla. org/rhino) implementation of Java. Script. n Management viewed as – Setting up the distributed application (Involves setting up the broker network, initializing system components) – Querying Run-time System Metadata § For logging purposes § Monitor metadata to help dynamically rewire system to improve performance May 10, 2005 Community Grids Lab Presentation for CCGrid 2005 3

HPSearch Architecture Component Summary n Binds URIs to a scripting language – Allows us to manage (manipulate) the Resource identified by the URI § E. g. Read / Write to files, sockets, topics § Read from data base, data from FTP/HTTP resource § This data can then be streamed to distributed application OR data can be read from a stream and processed / stored n “Host-objects” allow us to dynamically access the host system. – Useful for constructing objects that monitor system meta-data, perform management tasks – Perf. Metrics gathers system performance data and allows us to query it at run-time. May 10, 2005 Community Grids Lab Presentation for CCGrid 2005 4

HPSearch Architecture Diagram HPSearch Kernel Request Handler Java. Script Shell Request Handler Other Objects Data. Base JDB DBHandler Web Service TTP P/H SOA WSDLHandler WSProxy. Handler l oco t Pro C URIHandler Task Scheduler Flow Handler Web Service EP ork w Net Files Sockets Topics SOAP/HTTP WSProxy Service HPSearch Control Events using PUB/SUB on predefined topic Data buffers sent / received as Narada Events Broker Network WSProxy Service May 10, 2005 Community Grids Lab Presentation for CCGrid 2005 . . . HPSearch Kernel WSProxy Service 5

Narada. Brokering n Narada. Brokering – Messaging infrastructure for collaboration, peer-topeer and Grid applications – Implements high-performance protocols (message transit time of 1 to 2 ms per hop) – Order-preserving, optimized message transport with Qo. S and security profiles for sent and received messages – Support for different underlying protocols such as TCP, UDP, Multicast, RTP – Discovery Service to locate nearest brokers May 10, 2005 Community Grids Lab Presentation for CCGrid 2005 6

HPSearch + Narada. Brokering Managing Streams n HPSearch uses Narada. Brokering to route data streams – Each stream is represented by a topic name – Components subscribe / publish to specified topic § The WSProxy component automatically maps topics to Input / Output streams § Each write (byte[] buffer) and byte[] read() call is mapped to a Narada. Brokering event May 10, 2005 Community Grids Lab Presentation for CCGrid 2005 7

System Goals Investigate the use of HPSearch as a management console to deploy system and application components using scripting n Use HPSearch in scientific / grid applications which can use streaming data n – Data filtering is essential in most cases and we do not want to transfer entire data set when only a small percentage of data would be used, § E. g. Choosing data satisfying certain input constraints. – Data re-ordering might be required formatting input data to match the requirements of executable, § E. g. Converting GML to remove XML elements n Use publish-subscribe methodologies to connect components. May 10, 2005 Community Grids Lab Presentation for CCGrid 2005 8

Example RDAHMM GPS time series have modes caused by unknown underlying physical processes n RDAHMM allows us to identify these modes and time periods where physical processes dominated the sequence without any a-priori knowledge of these processes n Help to determine n – The actual physical causes – When is the system entering a new mode (perhaps a hint of some important seismic event) May 10, 2005 Community Grids Lab Presentation for CCGrid 2005 9

Example Application Components n n GPS database (Surface displacement time series collected by SCIGN, http: //www. scign. org) Data filter (filters data, removes unwanted components, reorders data as required by the actual RDAHMM executable) RDAHMM executable, performs the time series analysis. This data is transferred to the Graph plotting application. Matlab based Graph plotting application May 10, 2005 Community Grids Lab Presentation for CCGrid 2005 10

Applications Streaming Data Filtering GPS Data HPSearch Engine Sensor Source Handler HPSearch Handler Data Filters the input data to get only the estimate and error values Handler RDAHMM Analyze the data Matlab Plotting Script May 10, 2005 Each Handler controls operation of 1 service Output Graph Community Grids Lab Presentation for CCGrid 2005 11

Sample Output Claremont, CA May 10, 2005 Community Grids Lab Presentation for CCGrid 2005 12

Related Work n Scripting languages have been very popular and successful for Rapid Application Deployment (RAD) – Sash by IBM, allow RAD and handlers for various tasks such as reading from databases, LDAP registries, invoking Web Services and providing GUI – Jython and Matlab are popular in many scientific communities. E. g. Geo. DISE – XCAT project at IU, Extreme Lab uses Jython for scripting to deploy distributed components – Karajan (Ant Like scripting) is used for deploying applications over grid – WSRF: : Lite uses perl to implement Web Services Resource Framework and host grid services May 10, 2005 Community Grids Lab Presentation for CCGrid 2005 13

Future Work n n n Investigate system and management scaling with increasing number of components. Use of security for streams Negotiation of optimal (high performance) transport More handlers for different aspects of Narada. Brokering’s management (broker / topic discovery, security, replay etc…) Investigate interaction with WS Management and WS - Distributed Management based systems May 10, 2005 Community Grids Lab Presentation for CCGrid 2005 14

To conclude… n Presented a scripting based architecture for management of data streams and distributed services n Shown how we can use publish subscribe methodologies to connect components that process data in a stream n Create and deploy quick data filtering applications May 10, 2005 Community Grids Lab Presentation for CCGrid 2005 15

More Information n HPSearch Project Website http: //www. hpsearch. org n Narada. Brokering Project @ IU http: //www. naradabrokering. org n CGL Publications http: //grids. ucs. indiana. edu/ptliupages/publications/ May 10, 2005 Community Grids Lab Presentation for CCGrid 2005 16

Questions / Comments n Any Questions / Comments ? THANKS for attending the presentation May 10, 2005 Community Grids Lab Presentation for CCGrid 2005 17