LEAD VGr ADS Integration Lavanya Ramakrishnan Renaissance Computing

LEAD VGr. ADS Integration Lavanya Ramakrishnan Renaissance Computing Institute University of North Carolina – Chapel Hill Duke University North Carolina State University

Goals of the Workshop • • Develop a workplan for short-term LEADVGr. ADS integration — Brainstorm Research Solutions — Development and Integration — What can we do and by when? Outline some of the longer term interesting issues LEAD workflow provides for VGr. ADS

Recap Earlier Discussions • • Execute the static LEAD workflow under vg. ES with a deadline constraint What do we need? —Performance model for LEAD components, resource requirements —Scheduling - heuristic optimization, queue scheduling —Queue prediction —vg. ES

LEAD Workflow Architecture LEAD Portal Workflow, AWSDLs, Inputs (2) Workflow Description (Annotated DAG) Experimental Builder Portlet (1) Resource mapping WSDLs Workflow, WSDLs, Inputs (8) Workflow Engine Workflow Configuration Service (WCS) (4) Resource Provisioning Service (RPS) Service ID (5) WSDLs Service ID (6) WSDL Resource Info (7) WSDL Registry Service Generic Application Factory (GFac) Application Service

LEAD VGr. ADS Interaction Resource Provisioning Service (RPS) Workflow Description (DAG) + Constraints Application Scheduler Generated vg. DL vg. ES Virtual Grid Resource mapping (Annotated DAG) Performance Model Batch queue Prediction

LEAD Workflow Terrain data files NAM, RUC, GFS data 3 1 Terrain Preprocessor Radar data (level II) Radar data (level III) Satellite data 7 3 D Model Data Interpolator (Initial Boundary Conditions) IDV Bundle 4 Surface data, upper air mesonet data, wind profiler 88 D Radar Remapper 9 5 ADAS 6 8 WRF 12 WRF to ARPS Data Interpolator Surface, Terrestrial data files WRF Static Preprocessor 10 ARPS to WRF Data Interpolator NIDS Radar Remapper Satellite Data Remapper 2 11 (lateral Boundary Conditions) 13 Triggered if a storm is detected Run Once per forecast Region Repeated for periodically for new data Visualization on users request ARPS Plotting Program

LEAD Resource Requirements Terrain data files NAM, RUC, GFS data 3 1 Terrain Preprocessor 7 3 D Model Data Interpolator (Initial Boundary Conditions) 11 (lateral Boundary Conditions) Data mapping and processing Pre processing Step Radar data (level II) Radar data (level III) Single Processor Jobs data (few. Satellite minutes) 4 Surface data, upper air mesonet data, wind profiler Currently single node applications, could be made to be data parallel. Could be intensive. 5 Interpolator is MPI, ADAS might be NIDS Radar MPI in the future Remapper 88 D Radar Remapper ADAS 6 8 Satellite Data Remapper WRF Static Preprocessor 9 Processing 10 MPI applications WRF Large part of running time of the workflow ARPS to WRF Data Interpolator 12 WRF to ARPS Data Interpolator Surface, Terrestrial data files 2 IDV Bundle 13 Triggered if a storm is detected Run Once per forecast Region Repeated periodically for new data Visualization on users request ARPS Plotting Program

Agenda • April 18, 2006 • April 19, 2006 • April 20, 2006 • April 21, 2006 —Introduction, Big Picture discussion (Lavanya) —LEAD workflow logistics to feed to the workplan– machines, application codes, etc (Suresh, Lavanya) —LEAD-VGr. ADS interface points (Suresh) — Requirements, changes in vg. ES (Yan-Suk Kee) — Performance Model of the LEAD workflow (Lavanya) — 9: 00 am Batch queue prediction in vg. ES (Daniel Nurmi) — 1: 00 pm Scheduler Interactions (Ryan Zhang) — 9: 00 am Work plan, Milestones (Lavanya)

Comments?