What is the Matrix Arun swaran Jagatheesan San

What is the Matrix? Arun swaran Jagatheesan San Diego Supercomputer Center DAKS Meeting, 11/03/2003 SDSC National Partnership for Advanced Computational Infrastructure University of Florida San Diego Supercomputer Center

Credit / Acknowledgements • Participants • • • Allen Ding Lucas Gilbert Reena Mathew Erik Vandiekieft ( IBM) Xi Cynthia Sheng • Well Wishers • Reagan Moore & SRB Team • Kim Baldridge • YOU !!! • Sponsors • NSF Gri. Phy. N, NSF SCEC, NPACI REU, NIH BIRN National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center University of Florida 2

Talk Outline • • • Workflow Requirements for Grid Workflow Data Grid Language Matrix as a Wf. MS Demonstrations • XQuery (CDL) • External Status Requests National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center University of Florida 3

Workflow • Automation of business process • Whole or Part • Documents/Information or tasks passed between participants • Based on a set of procedural rules • Scientific Computing Workflow • Computational research process as pathways or pipelines • Gather data, cleanse data, apply different combinations of transformations, simulations, visualization, publish in digital library, archive data, get Nobel prize (makes us also happy : -) National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center University of Florida 4

Workflow Description • Workflow description needed for processing • Current standards • BPEL 4 WS (Gaining Popularity) • Business Process Execution Language for Web Services • WSFL, XLANG; Merger of IBM, MS, … • XPDL • XML Process Definition Language • From Workflow Management Coalition • UML, WSCI, … National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center University of Florida 5

Grid Workflow • Granular Metadata • Context-based workflow, with control-based constructs • Query • Dynamic rule (ECA rules) to update milestones • Runtime Query on status of other activities • XQuery subset - Uniform query for data and process • Looping • Scientific Computing more looping structures • Interest in execution of each iteration and change of state • No support from existing standards • http: //tmitwww. tm. tue. nl/research/patterns/standards. htm National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center University of Florida 6

Grid Workflow Description (Addl. Req. ) • • Facilitate modeling using a GUI as a workflow graph Describe rules that can be updated dynamically Embedded ECA-like rules within workflow description Describe looping structures - where the state of each iteration is visible to applications • Dynamic query description for inter and intra-process coordination during workflow • Context based workflow (not purely control based) • Support import and export of definitions to be executed in P 2 P environment of workflow systems (and also support composition of services) National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center University of Florida 7

Data Grid Language • XML based asynchronous protocol • Describe Workflow control structures (Sequence, Parallel Split, . . ) • Describe data sets, collections, datagrid operations, . . . • Access and manage data grids, data flow pipelines • Query on data resource (based on W 3 C XQuery subset) • Describe Rules and Context based Grid Workflow* • Query on Process meta-data, state* • Implementation • Reference Implementation by SDSC Matrix Project * Being Designed/developed as of the presentation date National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center University of Florida 8

Data Grid Request National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center University of Florida 10

Data Grid Response (DRes) National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center University of Florida 12

Grid Workflow Process I End User Workflow Description Data Grid Language National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center University of Florida 13

Grid Workflow Process II Abstract Workflow Data Grid Language Planner Concrete Workflow National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center University of Florida 14

Grid Workflow Process III Grid Workflow Processor Concrete Workflow Export Workflow to Matrix P 2 P National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center University of Florida 15

Matrix Server Acts as a Peer in Wf. MS P 2 P System * Processes Data Grid Requests Can maintain state an manage process steps Can invoke SRB data grid processes, OGSAServices, WSDL Services (OGSA Threads to be implemented) • Implemented as an Open-source Project • • * Being Designed/developed as of the presentation date National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center University of Florida 16

Implementation Status • Data Grid Language Schema for basic workflow constructs, Data Grid Operations • Matrix agents for executing data grid requests • Basic process pipeline management • Data Grid Language: Rules, Embedded query, OGSA operations to be added • Matrix: P 2 P, export/sharing of workflow to be added National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center University of Florida 17

SDSC Matrix Architecture SOAP Service Wrapper Abstraction JAXM Wrapper OGSA RPC-Style for SOAP Event Publish Subscribe, Notification JMS Messaging System Matrix Data Grid Request Processor Transaction Handler Flow Handler and Execution Manager Status Query Handler XQuery Processor Termination Handler Matrix Agent Abstraction Other SRB Data Agents Services OGSA Agent Pipeline Query Processor WSDL Agent Data flow pipeline Meta data Manager Persistence (Store) Abstraction JDBC In Memory Store National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center University of Florida 18

Conclusion • Data Grid Language • • Grid Workflow Description Basic Stuff or foundation ready Solid Design to handle more complex stuff Workflow Modeling not investigated (like Ptolemy? ) • Matrix Server Implementation • Create, Query, Manage Grid Workflows • OGSA, Rules, P 2 P to be implemented • More Support will expedite R&D National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center University of Florida 19

Demos ? He is trying to escape. Where are the Demos? National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center University of Florida 20