Workflow Group ACME Workbench Sterling Baldwin ACME Workflow
Workflow Group: ACME Workbench Sterling Baldwin ACME Workflow in the Browser Team Lead: Sterling Baldwin Sam Fries, Matt Harris, Bibi Raju, Lukasz Lacinski, Jim Mc. Enerney, Jeff Painter, Charles Doutriaux, Sahsa Ames
Project Goals • Entire ACME workflow in the browser
Project Goals • Entire ACME workflow in the browser – – – – Run the model Generate climatologies Run diagnostics Analyze and visualize output data Transfer files between facilities Archive to HPSS Publish to ESGF
Project Goals • Each part of the process works independently, and flows into the next step
Project Goals • Each part of the process works independently, and flows into the next step – Run the model -> Slurm/Moab and the ACME_script. csh
Project Goals • Each part of the process works independently, and flows into the next step – Run the model -> Slurm/Moab and the ACME_script. csh – Generate climatologies -> ncclimo
Project Goals • Each part of the process works independently, and flows into the next step – Run the model -> Slurm/Moab and the ACME_script. csh – Generate climatologies -> ncclimo – Run diagnostics -> metadiags
Project Goals • Each part of the process works independently, and flows into the next step – – Run the model -> Slurm/Moab and the ACME_script. csh Generate climatologies -> ncclimo Run diagnostics -> metadiags Analyze and visualize output -> Python and UVCDAT
Project Goals • Each part of the process works independently, and flows into the next step – – – Run the model -> Slurm/Moab and the ACME_script. csh Generate climatologies -> ncclimo Run diagnostics -> metadiags Analyze and visualize output -> Python and UVCDAT Transfer files between facilities -> Globus
Project Goals • Each part of the process works independently, and flows into the next step – – – Run the model -> Slurm/Moab and the ACME_script. csh Generate climatologies -> ncclimo Run diagnostics -> metadiags Analyze and visualize output -> Python and UVCDAT Transfer files between facilities -> Globus Archive to HPSS -> HTAR
Project Goals • Each part of the process works independently, and flows into the next step – – – – Run the model -> Slurm/Moab and the ACME_script. csh Generate climatologies -> ncclimo Run diagnostics -> metadiags Analyze and visualize output -> Python and UVCDAT Transfer files between facilities -> Globus Archive to HPSS -> HTAR Publish to ESGF -> esgf publication python module
Project Goals • Each part of the process works independently, and flows into the next step – – – – Run the model -> Slurm/Moab and the ACME_script. csh Generate climatologies -> ncclimo Run diagnostics -> metadiags Analyze and visualize output -> Python and UVCDAT Transfer files between facilities -> Globus Archive to HPSS -> HTAR Publish to ESGF -> esgf publication python module – That’s a lot of tools!
Workbench vision • Wrap each tool in a UI – – – Show the options for the tool (no manpage lookup required) Only allow correct input Automate where possible Each tool in one place Zero install on users machine
Workbench progress • Data Manager – Use Globus to transfer files from Edison – Import data from ESGF – Publish to our private ESGF staging node • Run Manager – Create, configure, and execute diagnostic jobs – Browse output – Upload to Diagnostic Viewer • Visualizations – Barebones VCS visualization of net. CDF files
Future work • ACME model run – Automatic post processing and data transfer, HPSS archive – Job chaining • job A triggers job B which triggers job C and D, or an error causes A to restart and email the error to the user. • Access controlled sharing for all user data – Share anything you want with just the people you want • Full visualization power of UV-CDAT – VCS in the browser
Demo
- Slides: 16