Workflow Management on Abel Abdulrahman Azab 5 Apr

Workflow Management on Abel Abdulrahman Azab 5, Apr

Kiss! Keep It Simple Stupid!

What is a Workflow? A series of activities that are necessary to complete a task.

What is a Workflow? B out. B. txt input. txt A D out. A. txt out. C. txt C output. txt

Workflow Control Login Node (abel. uio. no) B out. B. txt input. txt A D out. A. txt out. C. txt C out. D. txt Abel Controller

SDAG: Direct Acyclic Graph Workflow Submitter for SLURM

Direct Acyclic Graph (DAG) Directed graph with no directed cycles. Formed by a collection of vertices and directed edges. there is no way to start at some vertex and follow a sequence of edges that eventually loops back to the same vertex. No Loops

Direct Acyclic Graph (DAG)

Non-Direct Acyclic Graph (DAG)

SDAG Example JOB A job. A. sbatch JOB B job. B. sbatch JOB C job. C. sbatch PARENT A CHILD B PARENT B CHILD C A B C

SDAG Example JOB A job. A. sbatch JOB B job. B. sbatch JOB C job. C. sbatch JOB D job. D. sbatch B A D C PARENT A CHILD B C PARENT B C CHILD D

SDAG hands-on On Git. Hub: https: //github. com/abdulrahmanazab/sdag

SDAG hands-on Ø Ø Ø Login $git clone https: //github. com/abdulrahmanazab/sdag $cd sdag/example 1 $vim i 1 I want to go home : ( $vim i 2 I REALLY want to go home : ’(

SDAG hands-on python test. py i 1, i 2, … output 30 j. XX Interpreter Ø Ø code Input files output file Run-time $python test. py i 1, i 2 out 5 gift $cat out Job name

SDAG hands-on Ø Ø Ø $projects $vim job. A. sbatch #SBATCH --account=<your-project-name> The same with job. B. sbatch, job. C. sbatch, and job. D. sbatch B A D C

SDAG hands-on Ø Ø Ø $sbatch job. A. sbatch $squeue -u <your-username> $module load sdag $vim dag. sdag $sdag dag. sdag $squeue -u <your-username>