Working with Kepler Workflows to Automate Steps in

Working with Kepler Workflows to Automate Steps in the Scientific Method Ilkay ALTINTAS Director, Scientific Workflow Automation Technologies Laboratory San Diego Supercomputer Center, UCSD Ilkay ALTINTAS - July 24 th, 2007 1

The Big Picture: Supporting the Scientist From “Napkin Drawings” … … to Executable Workflows Conceptual SWF Executable SWF Source: Guide. To. ENM. pdf at the workshop web link. http: //reap. ecoinformatics. org/Wiki. jsp? page=Requirements. Workshop 2007 Ilkay ALTINTAS - July 24 th, 2007 2

Anatomy of a Kepler Workflow Toolbar Search Panel Director Atomic Actor Link Drag and Drop Annotation Ilkay ALTINTAS - July 24 th, 2007 3

Anatomy of a Kepler Workflow Composite Actor Atomic Actor Ilkay ALTINTAS - July 24 th, 2007 4

Anatomy of a Kepler Workflow Run Controls Parameter Ilkay ALTINTAS - July 24 th, 2007 5

Some False Myths about Scientific Workflows • Scientific workflows are… – … scripts/visual programs • Can be similar, but there’s value added – … just workflow engines • There are other components making the design and execution of workflows more effective • Search features, provenance tracking, semantic guidance, etc… – … the user interface • User interface is to make workflow design and manipulation easier Ilkay ALTINTAS - July 24 th, 2007 6

Kepler can be used as a batch execution engine. Ilkay ALTINTAS - July 24 th, 2007 7

A Three Tier Architecture for Li. DAR Interpolation and Analysis Efrat JEAGER-FRANK 1 Christopher J. CROSBY 2 Ashraf MEMON 1 Viswanath NANDIGAM 2 ARROWSMITH 2 Jeffrey CORNER 2 Ilkay ALTINTAS 1* Chaitan BARU 1 1 San J. Ramon Diego Supercomputer Center, University of California, San Diego 2 Department of Geological Sciences, Arizona State University Ilkay ALTINTAS - July 24 th, 2007 *Presenting author 8

R. Haugerud, U. S. G. S Survey Li. DAR Introduction Interpolate / Grid D. Harding, NASA Point Cloud x, y, zn, … Process & Classify Analyze / “Do Science” Ilkay ALTINTAS - July 24 th, 2007 9

Li. DAR Difficulties • Massive volumes of data – 1000 s of ASCII files – Hard to subset – Hard to distribute and interpolate • Analysis requires high performance computing • Traditionally: Popularity > Resources Ilkay ALTINTAS - July 24 th, 2007 10

A Three-Tier Architecture • GOAL: Efficient Li. DAR interpolation and analysis using GEON infrastructure and tools Portal – GEON Portal – Kepler Scientific Workflow System – GEON Grid • Use scientific workflows to glue/combine different tools and the infrastructure Grid Ilkay ALTINTAS - July 24 th, 2007 11

Lidar Workflow Process • • Configuration phase Subset: DB 2 query on Data. Star Portal Monitoring/ Translation Visualize Analyze Subset move process move render display • Interpolate: Grass RST, Grass IDW, GMT… • Visualize: Global Mapper, Fleder. Maus, Arc. IMS Scheduling/ Output Processing Grid Ilkay ALTINTAS - July 24 th, 2007 12

Lidar Processing Workflow (using Fledermaus) Visualize Analyze Subset move process move render display Arizona Cluster Fledermaus Create Scene file Datastar sd i. View 3 D/ Browser d 2 d 1 IBM DB 2 NFS Mounted Disk d 1 d 2 (grid file) d 1 d 2 NFS Mounted Disk Ilkay ALTINTAS - July 24 th, 2007 13

Lidar Processing Workflow (using Global Mapper) Visualize Analyze Subset move process move render display Arizona Cluster Global Mapper Get image for grid file Datastar Browser d 2 d 1 IBM DB 2 NFS Mounted Disk d 1 d 2 (grid file) d 1 d 2 NFS Mounted Disk Ilkay ALTINTAS - July 24 th, 2007 14

Lidar Processing Workflow (using Arc. IMS) Visualize Analyze Subset move process move render display Arizona Cluster Arc. IMS Datastar d 1 IBM DB 2 NFS Mounted Disk Arc. Info d 1 Arc. SDE Arc. IMS d 2 (grid file) d 1 d 2 NFS Mounted Disk Ilkay ALTINTAS - July 24 th, 2007 15

Lidar Workflow Portlet • User selections from GUI – Translated into a query and a parameter file – Uploaded to remote machine • Workflow description created on the fly • Workflow response redirected back to portlet Ilkay ALTINTAS - July 24 th, 2007 16

LIDAR POST-PROCESSING WORKFLOW PORTLET x, y, z and attribute Client/ NFS Mounted Disk DB 2 GEON Portal Render Map Parameters Grass Functions Arc. SDE Arc. Info Parameter xml Arc. IMS process output Create Workflow Description raw data DB 2 Spatial query submit Map onto the grid (Pegasus) Grass surfacing algorithms: Spline IDW Compute Cluster block mean Binary grid … ASCII grid Text file ASCII grid Tiff/Jpeg/Gif Download data KEPLER WORKFLOW Ilkay ALTINTAS - July 24 th, 2007 17

Portlet User Interface - Main Page Ilkay ALTINTAS - July 24 th, 2007 18

Portlet User Interface - Parameter Entry 1 Ilkay ALTINTAS - July 24 th, 2007 19

Portlet User Interface - Parameter Entry 2 Ilkay ALTINTAS - July 24 th, 2007 20

Portlet User Interface - Parameter Entry 3 Ilkay ALTINTAS - July 24 th, 2007 21

Behind the Scenes: Workflow Template Ilkay ALTINTAS - July 24 th, 2007 22

Filled Template Ilkay ALTINTAS - July 24 th, 2007 23

Example Outputs Ilkay ALTINTAS - July 24 th, 2007 24

With Additional Algorithms Ilkay ALTINTAS - July 24 th, 2007 25

ORB Sources: Real Time Sensor Data Source. KEPLER ANTELOPE Orb Data Recorded. CLIENT • Start Kepler Engine via a Browser • Specify Contour Parameters • Look at Contour Plots with Google Maps Applications Ilkay ALTINTAS - July 24 th, 2007 WEB SERVER 26

To Sum Up • The next three days: – Discussions on use cases with an eye on the workflow requirements – Discuss other possible workflows – Developers taking notes on Kepler implementation details of possible workflow steps • Latest release: http: //kepler-project. org Ilkay ALTINTAS - July 24 th, 2007 27

Thanks! & Questions… Ilkay Altintas altintas@sdsc. edu +1 (858) 822 -5453 http: //www. sdsc. edu Ilkay ALTINTAS - July 24 th, 2007 28