DGL The Assembly Language for Grid Computing Arun
DGL: The Assembly Language for Grid Computing Arun swaran Jagatheesan arun@sdsc. edu San Diego Supercomputer Center (SDSC) University of California, San Diego 13 th IEEE International Symposium on High-Performance Distributed Computing (HPDC) June 4 -6, 2004, Honolulu, Hawaii, USA Grid Physics Network (Gri. Phy. N) University of Florida San Diego Supercomputer Center
Acknowledgement • Participants • • Jonathan Weinberg Allen Ding Dipti Borkar Lucas Gilbert (BIRN) • Alumnii • Erik Vandekieft • Reena Mathew • Good-will Wishers • Reagan Moore and SDSC SRB Team • Marcio Faerman (SCEC) • You !!! Grid Physics Network (Gri. Phy. N) University of Florida 2 San Diego Supercomputer Center
Talk Outline • • • Data Grids in production for Scientific pipelines Problem : Gridflow Description and Querying Gridflows Description Requirements Relevance to P 2 P Gridflow Management Gridflow Data Model and components Summary Grid Physics Network (Gri. Phy. N) University of Florida 3 San Diego Supercomputer Center
Data Grid Management Systems Southern California Earthquake Center NASA Data Grids All these Data Grids have pipelines for their scientific or business processes NIH Biomedical Informatics Research Network National Science Digital Library Scripps Institute of Oceanography Grid Physics Network (Gri. Phy. N) University of Florida 4 San Diego Supercomputer Center
Data handling pipeline in SCEC (data information pipeline) Metadata derivation Ingest Data Ingest Metadata Pipeline could be triggered by input at data source or by a data request from user Determine analysis pipeline Initiate automated analysis Use the optimal set of resources based on the task – on demand Organize result data into distributed data grid collections All gridflow activities stored for data flow Grid Physics Network (Gri. Phy. N) provenance University of Florida 5 San Diego Supercomputer Center
Data Discovery New data Digital entities updates relationships among data in collections Meta-data Services invoked to analyze new relationships Services DGMS applications get notified of state updates State Grid Physics Network (Gri. Phy. N) University of Florida 6 San Diego Supercomputer Center
What they want? We know the business (scientific) process Cyber. Infrastructure is all we care (why bother about atoms or DNA) Grid Physics Network (Gri. Phy. N) University of Florida 7 San Diego Supercomputer Center
What they want? Use DGL to describe your process logic with abstract references to datagrid infrastructure dependencies Grid Physics Network (Gri. Phy. N) University of Florida 8 San Diego Supercomputer Center
Gridflows • Grid Workflow (Gridflow) is the automation of a execution pipeline in which data or tasks are processed through multiple autonomous grid resources according to a set of procedural rules • Gridflows are executed on resources that are dynamically obtained through confluence of one or more autonomous administrative domains (peers) Grid Physics Network (Gri. Phy. N) University of Florida 9 San Diego Supercomputer Center
Gridflow Language and CS Domains • Compiler Design • Variable scope definition, Recursive Grammar, Execution Stack Management, • Data Modeling • Schema definitions for gridflow patterns • Grid Computing • Data Grid data types, Virtual Organization, basic operations, … • Other concepts and Standards • Rules, W 3 C XQuery, GGF JSDL? Grid Physics Network (Gri. Phy. N) University of Florida 10 San Diego Supercomputer Center
Gridflow Language Requirements • High level Abstract descriptions • Abstract description of cyberinfrastructure dependencies • Simple yet flexible • Flexible to describe complex requirements (no brute force) • Gridflow dependency patterns • Based on execution structure and data semantics • (Parallel, Sequential, fork-new), (milestones, for-each, switch-case). . • Asynchronous execution • For long-run requests • Querying using existing standard • XQuery Grid Physics Network (Gri. Phy. N) University of Florida 11 San Diego Supercomputer Center
Gridflow Language Requirements • Process meta data and annotations • Runtime definition, update and querying of meta-data • Runtime Management of Gridflows • Stop gridflow at run time • Partitioning • Facility in language to divide a gridflow request to multiple requests • Import descriptions • Refer other gridflows in execution Grid Physics Network (Gri. Phy. N) University of Florida 12 San Diego Supercomputer Center
Data Grid Language (DGL) • DGL is just a language specification • Can be used in any commercial or academic data grid software • DGL describes gridflow description and dependencies Grid Physics Network (Gri. Phy. N) University of Florida 13 San Diego Supercomputer Center
Gridflow Process I End User using DGBuilder Grid Physics Network (Gri. Phy. N) University of Florida 14 Gridflow Description Data Grid Language San Diego Supercomputer Center
Gridflow Process II Abstract Gridflow using Data Grid Language Grid Physics Network (Gri. Phy. N) University of Florida 15 Planner Concrete Gridflow Using Data Grid Language San Diego Supercomputer Center
Gridflow Process III Gridflow Processor Concrete Gridflow Using Data Grid Language Grid Physics Network (Gri. Phy. N) University of Florida 16 Gridflow P 2 P Network San Diego Supercomputer Center
DGL Structure (data model) Flow Logic Structure Pre-Process Structure – parallel, sequential etc. , ECA Rule based definitions Runnable Recursive definition of runnables as either data operation or as a executable process (Job) Post-Process Grid Physics Network (Gri. Phy. N) University of Florida 17 Meta-data San Diego Supercomputer Center
Operations in DGL • Execute Process (DAG, java, WSDL, etc) • Very generic Datagrid operations • • Copy directories/files Change Permissions (Chmod) Create directory/file/archive Delete directory/file/archive Ingest/download URl or any data source Replicate, Rename, List Seek. NWrite, Seek. NRead Ingest, Query Any type of Metadata Grid Physics Network (Gri. Phy. N) University of Florida 18 San Diego Supercomputer Center
Components of DGL • DGL document is either a request or a response • Data Grid Request • Could be a Flow (aggregation of operations) • Or could be a Status Query • Data Grid Response • Could be a Flow Acknowledgement • Or could be a Status Response • Can be made Synchronous or Asynchronous • Flexibility for any type of Implementation Grid Physics Network (Gri. Phy. N) University of Florida 19 San Diego Supercomputer Center
Summary • A standard description language is Needed • Requirements of the language • Data Grid Language (DGL) • • Recursive definition of flows and steps Metadata or variable scopes Rules Can be partitioned (sub-divided) • Components of Data Grid Language • Next step: Talk to Scheduling or Heuristics people Grid Physics Network (Gri. Phy. N) University of Florida 20 San Diego Supercomputer Center
got ideas/suggestions? Contact: SDSC Matrix project arun@sdsc. edu Google key word: SDSC Gridflow Click here to start the slide show again Grid Physics Network (Gri. Phy. N) University of Florida San Diego Supercomputer Center
- Slides: 21